- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents DuelBench, a novel benchmark designed to evaluate LLM agents in closed-loop web attack and defense interactive scenarios on live services.
Critical strengths
Unlike existing benchmarks that evaluate LLM agent exploits against static target defenses, DuelBench introduces live defender agents that dynamically inspect logs and update mitigation rules during an attack.
Critical weaknesses
The target service audit logs explicitly label incoming requests with attacker and benign tags. In real-world security monitoring, a defender's primary challenge is identifying adversarial traffic amidst normal user noise without prior ground-truth tags.
Questions for authors
-
The proposed framework is suspected to be generated by AI. The author needs to declare whether the proposed framework uses AI graphics. The author only stated in the statement that AI is used to polish and write code, and I am not sure if this includes the use of AI for drawing frameworks.
-
The results show that short-horizon attacks maintain near 100% ASR across all defense conditions. Does this structural delay mean that DuelBench is currently unable to evaluate defensive capabilities against single-step or short-horizon vulnerabilities?
-
How do the authors justify the inclusion of these short scenarios in the overall benchmark average if the defensive agent has no opportunity to intervene?
-
In realistic cybersecurity settings, a defender's primary challenge is detecting anomalies and attributing threats from noisy, unlabeled logs. How significantly does providing perfect ground-truth source tags artificially boost the performance of defender agents?
Thank you for your detailed feedback on our manuscript. Below, we respond to each of your points.
Source labels indicate which past requests came from the attacker, but they do not tell the defender which blocking rule will be effective. For example, blocking a shared page may stop the attack but also prevent normal users from accessing it. Blocking the attacker's IP keeps the page available, but the attacker may switch to another IP.
Therefore, the defender still needs to choose an effective response. Some studies [R1][R2][R3] also evaluate agents on fixing known vulnerabilities. They provide vulnerability details and test whether the agent can fix the issue.
To further address your concern, we test defense without source labels and discuss the results in our response to Q4 below.
[R1] Andy K. Zhang, et al., BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems. NeurIPS 2025.
[R2] Hwiwon Lee, et al., SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks. NeurIPS 2025.
[R3] Zheng Yu, et al., PatchAgent: A Practical Program Repair Agent Mimicking Human Expertise. USENIX Security 2025.
We used AI to generate the two robot icons in the framework figure. The other icons come from publicly available online resources. We created the framework's structure and layout manually in PowerPoint and exported it as a PDF.
We already selected “Yes, to aid or polish writing” under AI Assistance in the submission form. We considered the use of AI for these icons to fall within this category. In the revised version, we have clarified this use of AI in the AI Use Statement section.
DuelBench can evaluate defense against short attacks that leave time to respond. A successful first-turn attack leaves no such chance under the schedule in Section 3.2. Its ASR therefore does not measure the defender's ability to stop it.
Besides, we tested a DeepSeek-V4-Pro attacker against a Qwen3.6-Flash defender with labels and k=1. We ran each of the four short-attack scenarios 10 times. All attacks succeeded.
Therefore, the high ASR reflects both a lack of response time and unsuccessful defense. We have added this analysis to Appendix D.5 of the revised version.
The averages used for our main comparisons already exclude short scenarios. Tables 3 and 5 report the multi-step mean, and Figure 3 averages the same five multi-step scenarios. Appendix D.3 defines this averaging rule. Short scenarios are reported separately to show the limits of response timing.
We have made this point more explicit in the captions of Tables 3 and 5 and Figure 3 in the revised version.
We run an additional comparison with a DeepSeek-V4-Pro attacker and k=1. Each scenario has ten runs. We compare logs with and without labels, keeping all other settings the same.
The effect of source labels varies across defenders. With labels, Qwen allows fewer attacks to succeed, especially on multi-step tasks. DeepSeek's ASR changes little, but it blocks fewer normal requests. Without labels, both defenders still reduce overall ASR compared with no defense.
We have added this comparison to Appendix E.2 of the revised version.