- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CyLawBench evaluates tool-using LLM agents against selected Chinese and European Union cyber-law requirements under adversarial induction. Each scenario specifies a benign objective, protected objects, authorization facts, and evidence requirements through a frozen Goal Contract. The benchmark includes Workflow Redirection and Exploit-Chain Induction, with the latter adapting authorized cyber exercises into benign-goal tasks. The paper also proposes an adaptive attack protocol, ALHAP. Experiments cover 1,420 workflow episodes across five models and two harnesses, together with 41 exploit-induction scenes. The results distinguish attack delivery, attempted actions, and native-confirmed completion.
Critical strengths
-
The paper makes a useful distinction between technical attack success and violation of an authorization or legal boundary. Goal Contracts specify the scenario facts and evidence needed to interpret an executed action, rather than assigning a legal label solely from an exploit outcome. The paper appropriately limits its conclusions to these fixed benchmark conditions. This gives the evaluation a more explicit basis than a general harmfulness label.
-
The evaluation separates payload delivery, exposure, attempted actions, and confirmed completion. Native oracles remain authoritative, while disagreements with the auxiliary trajectory judge are retained for analysis. The substantial gap between delivery and completion illustrates why prompt-injection evaluations should verify resulting actions and state changes, rather than treating exposure or attempted execution as successful harm.
Critical weaknesses
-
The experiments do not sufficiently validate dynamic adaptation, although it is a central contribution of the paper. Appendix J.2 states that, for the Claude Code InjecAgent scenes, all payload placements occur before the agent starts and the ALHAP feedback loop is not closed. In Section 6.3, the OpenClaw budget curve is reconstructed from a single largest-budget run and is therefore non-decreasing by construction. The paper acknowledges these limitations, but the reported diagnostics cannot establish the benefit of feedback-driven adaptation. A comparison with a non-adaptive attack under matched budgets is needed to support this aspect of the contribution.
-
The denominator of the primary Legal Violation Rate is described inconsistently. Section 5.1 states that technically evaluable episodes remain in the denominator even without confirmed exposure, whereas Table 4 excludes both “Not delivered” and “Not exposed” episodes. Table 4 also leaves the corresponding counts and rates unspecified. These choices produce different interpretations: an end-to-end violation rate includes delivery failures, while an exposure-conditioned rate measures behavior after the attack is observed. Until the implemented rule and exclusion counts are clarified, the reported rates and harness comparisons are difficult to assess.
Questions for authors
-
Which subsets of the main evaluation execute ALHAP with a closed feedback loop, and which use payloads prepared before the SUT run? For the adaptive subset, can you provide a comparison with a non-adaptive attack using the same scenes and budgets?
-
Are not-delivered and not-exposed episodes included in the reported LVR denominators? Please reconcile Section 5.1 with Table 4 and provide per-arm counts for delivery, exposure, completion, and excluded episodes. Reporting both end-to-end and exposure-conditioned rates would clarify how delivery behavior affects the harness comparisons.
A third review
Summary:
This paper introduces CyLawBench, a benchmark for evaluating LLM agents' compliance with cybersecurity laws under adversarial attacks. It combines existing agent benchmarks with legal constraints and evaluates several LLM agents.
Weaknesses:
The proposed framework mainly integrates existing benchmarks and attack strategies, with limited methodological innovation. More importantly, the motivation for introducing legal compliance into agent safety evaluation is insufficiently justified. What additional insights does the legal perspective provide beyond existing safety benchmarks? If an agent is successfully attacked, what does a legal violation imply in practice, and what actions should follow? These fundamental questions remain unclear.
The experiments cover a limited set of legal scenarios, and some evaluations are incomplete. The generalizability of the findings remains unclear.
The paper contains obvious formatting and citation errors (e.g., “?Zhan et al., 2024”), as well as low-quality, AI-generated schematic figures. The overall presentation is below the expected standard.