- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes to use "vulnerability concepts" to improve replicability and verifiability of agent cybersecurity attacks. The idea of "Agent Hacks Agents" (AHA) is to discover these concepts and then test falsifiable hypotheses. They test this on AgentHazard, Agent-Dyn, and DTap benchmarks, and show that AHA has a higher attack success rate.
Critical strengths
Replication and vulnerability of agent based cybersecurity attacks is an important topic. The vulnerability concept can be useful for this.
Critical weaknesses
I am confused about the main comparison in Table 1. The paper states that "Baselines generate discovery attacks with Qwen-3.7-Max", but Section 4.1 states that AHA is run with Claude Opus-4.8 and Codex (GPT 5.5). That seems to confound the main comparison. Table 2 keeps the same backbone (Qwen-3.7-Max) but then the result is not different from IterInject.
The autoresearch loop relies extensively on LLMs, so falsification here is a bit funny.
The paper should report statistical significance tests for results.
While patches lower the attack rate, one of the most difficult things about patching is verifying that other functionality is not broken.
Questions for authors
Can you address the weaknesses noted above?
3 comments
Response to Reviewer Js2s (Part 2/2)
Q2: "The autoresearch loop relies extensively on LLMs, so falsification here is a bit funny."
A2: Falsification relies on external trajectory evidence rather than self-evaluation: pre-committed falsifiers, independent benchmark judges, and periodic fresh-context Critic audits guard against self-grading (Table 3), with 89.0% human agreement on 100 sampled reflections:
Js2s-Table 2. Agreement between the Reflector and two annotators on 100 reflections.
| Reflector's label | Reflections | Annotators agree with each other (%) | Consensus agrees with the Reflector (%) |
|---|---|---|---|
| Confirmed | 59 | 84.75 | 93.22 |
| Falsified | 32 | 93.75 | 87.50 |
| Inconclusive | 9 | 88.89 | 66.67 |
| All | 100 | 88.00 | 89.00 |
Results analysis. The consensus of the two annotators agrees with the Reflector on 89 of the 100 reflections, with Cohen's κ = 0.78. Together with the pre-committed falsifier, the independent outcome judge, and the periodic Critic, this ensures that falsification rests on verifiable trajectory evidence.
Q3: "The paper should report statistical significance tests for results."
A3: Statistical significance tests confirm that AHA's improvements over baselines are highly significant. Across all 18 benchmark settings (Table 1), paired -tests against the strongest baseline in each cell yield (), and Wilcoxon signed-rank tests yield (). Compared directly against AutoRISE*, a paired -test gives (), with 1,000-sample bootstrap 95% confidence intervals of [37.83%, 56.11%] for AHA versus [23.12%, 42.48%] for AutoRISE*. Furthermore, with the Qwen-3.7-Max researcher across all nine Claude Code settings, AHA's advantage is statistically significant against each of the five baselines (paired from 2.87 to 6.03, all ), including AutoDAN-Turbo* (mean 42.39% vs. 33.60%, paired , ) and AutoRISE* running on Claude-Opus-4.8 (mean 42.39% vs. 36.49%, paired , ).
Q4: "While patches lower the attack rate, one of the most difficult things about patching is verifying that other functionality is not broken."
A4: We add a new analysis and experiment about benign user task. The patch bundle keeps four of five benign completions while removing 41.11 points of attack success. On all 60 benign AgentDyn user tasks without injections, the patch bundle blocks 9.72% of tasks and benign completion moves from 49.17% to 39.44%, while harmful ASR on AgentHazard falls by 41.11 points (added to Section 5.4):
Js2s-Table 3. Benign AgentDyn user tasks with and without the patch bundle.
| Harness | Victim model | Benign tasks | Completed w/o bundle (%) | Completed w/ bundle (%) | Blocked by bundle (%) |
|---|---|---|---|---|---|
| Claude Code | Minimax-M2.7 | 60 | 45.00 | 36.67 | 8.33 |
| Claude Code | Kimi-K2.6 | 60 | 53.33 | 45.00 | 8.33 |
| Claude Code | Deepseek-V4-Pro | 60 | 61.67 | 51.67 | 10.00 |
| Codex | Minimax-M2.7 | 60 | 40.00 | 28.33 | 11.67 |
| Codex | Kimi-K2.6 | 60 | 43.33 | 35.00 | 8.33 |
| Codex | Deepseek-V4-Pro | 60 | 51.67 | 40.00 | 11.67 |
Results analysis. The bundle costs 9.72 points of benign completion and removes 41.11 points of attack success. Four of five benign tasks that completed before the bundle still complete with it (39.44% against 49.17%). The cost is also even across settings: the bundle blocks 8.33% to 11.67% of benign tasks under every harness and victim model.
We hope these additions address the reviewer's main concerns, and we would be glad to provide further details.
Follow-up to Reviewer Js2s
Dear Reviewer Js2s,
We are writing to ask whether our response has addressed your concerns. We would be happy to discuss any remaining issues.
Best regards,
Authors of Submission #928
Response to Reviewer Js2s (Part 1/2)
We sincerely thank the reviewer for the careful review. We are encouraged that the reviewer finds the replication and vulnerability of agent-based attacks an important topic and finds the vulnerability concept useful for it. We respond to the four concerns in order. The revised manuscript (PDF) is updated, with all changes highlighted in teal.
Q1: "I am confused about the main comparison in Table 1. The paper states that 'Baselines generate discovery attacks with Qwen-3.7-Max', but Section 4.1 states that AHA is run with Claude Opus-4.8 and Codex (GPT 5.5). That seems to confound the main comparison. Table 2 keeps the same backbone (Qwen-3.7-Max) but then the result is not different from IterInject."
A1: Held-out evaluation uses a single host model across all methods. At held-out time, one common host model instantiates every stored artifact and runs AutoRISE* (Appendix B.6). The other four baselines cannot run their attack generation on Claude-Opus-4.8 or GPT-5.5, so they generate with Qwen-3.7-Max, while AutoRISE* runs on the same host model as AHA (Appendix B.6). Extending Qwen-3.7-Max discovery to all nine Claude Code settings achieves 42.39% ASR with 688 queries (added to Section 5.1 and Appendix C.2).
Js2s-Table 1. Held-out ASR (%) under Claude Code on the Minimax-M2.7, Kimi-K2.6, and Deepseek-V4-Pro victims, with each method's researcher model, and total discovery queries Q over the nine settings. The rows marked Qwen-3.7-Max share one researcher model. AutoRISE* runs on Claude-Opus-4.8.
Results analysis. The tie with IterInject* is one setting of nine. On the nine-setting mean, AHA with Qwen-3.7-Max outperforms every baseline by 5.90 to 15.83 points with the fewest discovery queries. It is the best or tied for the best in seven of the nine settings, and stays within 3.06 points of the best baseline on the other two.