acceptodds
Desk rejected at ICLR 2027

CyLawBench: Evaluating LLM Agents' Compliance with Cyber Law under Dynamic Adversarial Induction

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Tool-using LLM agents increasingly act on networked systems, yet attack success and technical capability alone do not establish violations of authorization boundaries or legally protected interests. We present CyLawBench, an evidence-centered benchmark assessing agent cybersecurity against selected Chinese and European Union cyber-law provisions. Each scenario uses a frozen Goal Contract specifying benign goals, protected objects, authorization facts, and rule-specific evidence requirements. CyLawBench comprises Workflow Redirection, testing unauthorized operations in ordinary tool-use workflows, and Exploit-Chain Induction, testing whether benign-goal agents can be induced to execute exploit sequences. Its attack protocol supports trajectory-guided adaptation of indirect prompt injections while preserving scenario contracts. We also introduce offensive inversion, reconstructing authorized cyberattack exercises around benign objectives while retaining native environments and technical oracles. Our evaluation covers 1,420 workflow episodes across five models and two harnesses, plus 41 transformed exploit-induction scenes across five models and two guidance conditions. Among technically evaluable episodes, pooled Legal Violation Rates reach 19.1% for Workflow Redirection and 14.3% for Exploit-Chain Induction under Claude Code, with model rankings varying across tasks and harnesses. Gaps between delivery, attempted actions, and native completion highlight the need to ground benchmark-scoped legal judgments in contract-specific outcome evidence rather than exposure or technical capability alone.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
CyberSecurity

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_jW4JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CyLawBench evaluates tool-using LLM agents against selected Chinese and European Union cyber-law requirements under adversarial induction. Each scenario specifies a benign objective, protected objects, authorization facts, and evidence requirements through a frozen Goal Contract. The benchmark includes Workflow Redirection and Exploit-Chain Induction, with the latter adapting authorized cyber exercises into benign-goal tasks. The paper also proposes an adaptive attack protocol, ALHAP. Experiments cover 1,420 workflow episodes across five models and two harnesses, together with 41 exploit-induction scenes. The results distinguish attack delivery, attempted actions, and native-confirmed completion.

Critical strengths
  1. The paper makes a useful distinction between technical attack success and violation of an authorization or legal boundary. Goal Contracts specify the scenario facts and evidence needed to interpret an executed action, rather than assigning a legal label solely from an exploit outcome. The paper appropriately limits its conclusions to these fixed benchmark conditions. This gives the evaluation a more explicit basis than a general harmfulness label.

  2. The evaluation separates payload delivery, exposure, attempted actions, and confirmed completion. Native oracles remain authoritative, while disagreements with the auxiliary trajectory judge are retained for analysis. The substantial gap between delivery and completion illustrates why prompt-injection evaluations should verify resulting actions and state changes, rather than treating exposure or attempted execution as successful harm.

Critical weaknesses
  1. The experiments do not sufficiently validate dynamic adaptation, although it is a central contribution of the paper. Appendix J.2 states that, for the Claude Code InjecAgent scenes, all payload placements occur before the agent starts and the ALHAP feedback loop is not closed. In Section 6.3, the OpenClaw budget curve is reconstructed from a single largest-budget run and is therefore non-decreasing by construction. The paper acknowledges these limitations, but the reported diagnostics cannot establish the benefit of feedback-driven adaptation. A comparison with a non-adaptive attack under matched budgets is needed to support this aspect of the contribution.

  2. The denominator of the primary Legal Violation Rate is described inconsistently. Section 5.1 states that technically evaluable episodes remain in the denominator even without confirmed exposure, whereas Table 4 excludes both “Not delivered” and “Not exposed” episodes. Table 4 also leaves the corresponding counts and rates unspecified. These choices produce different interpretations: an end-to-end violation rate includes delivery failures, while an exposure-conditioned rate measures behavior after the attack is observed. Until the implemented rule and exclusion counts are clarified, the reported rates and harness comparisons are difficult to assess.

Questions for authors
  1. Which subsets of the main evaluation execute ALHAP with a closed feedback loop, and which use payloads prepared before the SUT run? For the adaptive subset, can you provide a comparison with a non-adaptive attack using the same scenes and budgets?

  2. Are not-delivered and not-exposed episodes included in the reported LVR denominators? Please reconcile Section 5.1 with Table 4 and provide per-arm counts for delivery, exposure, completion, and excluded episodes. Reporting both end-to-end and exposure-conditioned rates would clarify how delivery behavior affects the harness comparisons.

Reviewer_usa1OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the CyLawBench, which evaluates whether tool-using LLM agents can be induced to perform actions that violate selected Chinese and EU cyber-law provisions under predefined scenario facts. The benchmark covers ordinary tool-use workflows and exploit-induction tasks adapted from existing cybersecurity environments. Each scene has a frozen Goal Contract that specifies the legitimate task, authorization boundary, target actions, and required evidence. The paper also proposes ALHAP, an attack protocol that uses trajectory feedback to adjust injections. The experiments compare five models across two harnesses and track attack delivery, attempted actions, and completed harmful sequences separately.

Critical strengths
  1. The question studied is interesting. As agents gain access to tools and services, evaluating whether they respect authorization boundaries becomes increasingly relevant.
  2. Tracking delivery, exposure, attempted actions, and completed actions separately is useful. It helps readers see where an attack stopped, and the use of native execution evidence makes the results easier to inspect.
  3. I found offensive inversion interesting. Reusing an existing exploit environment while giving the agent a benign task could provide useful test cases. The authors are also careful about the scope of their legal claims and acknowledge several limitations of the experiments
Critical weaknesses
  1. My main concern is that I could not assess how well the legal mappings have been validated. For the retained scenes, a completed harmful sequence counts as a legal violation by construction. LVR therefore depends heavily on whether that mapping is sound. The paper lists the relevant provisions and describes the contract structure, but I wanted to see at least one example worked through in full: which facts establish applicability, which actions satisfy the rule, and what evidence supports each finding.

It is also unclear from the main text who reviewed these mappings and how disagreements were handled. The reported agreement between the trajectory Judge and native oracles does not answer this question, since agreement about task completion does not validate the underlying legal interpretation. Without this information, I found it difficult to judge the paper’s central contribution.

  1. I feel that the experiments do not sufficiently demonstrate the value of adaptive attacks. ALHAP receives considerable attention, but there is no comparison with an attacker using the same budget without feedback or strategy adaptation. The budget experiment does not fill this gap: Section 6.3 says that the Claude Code runs lack a closed online feedback loop for those scenes, while the OpenClaw results come from prefixes of a largest-budget run.

  2. The interpretation of low violation rates remains a problem. A model can obtain a low LVR because it rejects the malicious instruction, cannot complete the exploit, fails the legitimate task, or encounters an execution problem.

Questions for authors
  1. Could the authors walk through one workflow scene and one exploit-induction scene, including the legal mapping, facts visible to the agent, evidence predicates, and final judgment? How were these mappings reviewed?

  2. Which reported experiments use online trajectory feedback? How does ALHAP perform against a non-adaptive attacker with the same budget and access to injection surfaces?

Comments on the paper

1 comment
Area_Chair_VDE5OpenReview ↗

A third review

Summary:

This paper introduces CyLawBench, a benchmark for evaluating LLM agents' compliance with cybersecurity laws under adversarial attacks. It combines existing agent benchmarks with legal constraints and evaluates several LLM agents.

Weaknesses:

  1. The proposed framework mainly integrates existing benchmarks and attack strategies, with limited methodological innovation. More importantly, the motivation for introducing legal compliance into agent safety evaluation is insufficiently justified. What additional insights does the legal perspective provide beyond existing safety benchmarks? If an agent is successfully attacked, what does a legal violation imply in practice, and what actions should follow? These fundamental questions remain unclear.

  2. The experiments cover a limited set of legal scenarios, and some evaluations are incomplete. The generalizability of the findings remains unclear.

  3. The paper contains obvious formatting and citation errors (e.g., “?Zhan et al., 2024”), as well as low-quality, AI-generated schematic figures. The overall presentation is below the expected standard.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.