- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CryptoDojo, a framework for evaluating cryptocurrency agents under adversarial changes to transaction execution. Its central premise is that individually authorized actions can violate user policies due to changes in financial state, transaction ordering, persistent authority, or interactions across workflows. The authors propose the Temporal-Compositional Attack Model (T-CAM) and construct a benchmark of 200 paired ordinary/adversarial instances. Across four LLM-based agent configurations, safe completion drops from 53.5-90.5% under ordinary conditions to 0.5–14.0% under attack. The results also highlight that low policy-violation rates do not necessarily imply high task completion.
Critical strengths
-
Execution-aware evaluation of financial safety and determinisitc criteria. CryptoDojo evaluates actual financial outcomes rather than relying solely on agents' decisions or explanations. Its separation of task completion, policy violation, and safe completion is particularly useful for identifying transactions that settle successfully but violate timing, authorization, or financial constraints.
-
Paired evaluation reveals meaningful safety-utility tradeoffs. The ordinary/adversarial pairing controls for tasks and initial conditions, while the experimental results demonstrate substantial differences between agents' completion and violation rates.
Critical weaknesses
-
Unclear whether the benchmark measures agent safety or limitations of the execution environment. Some attacks deliberately introduce state changes between an agent's observation and transaction commitment. Without mechanisms for observing such changes or enforcing constraints atomically, an agent cannot reliably prevent these violations, regardless of its reasoning capabilities.
-
Lack of evaluated defenses or mitigation strategies. Although CryptoDojo is designed to support varying defenses, the experimental evaluation considers only four agent configurations without explicit mitigation mechanisms or systematic defense comparisons.
Questions for authors
See weaknesses, and:
- Unclear crypto-specific novelty. Many of the evaluated failure mechanisms, including stale-state decisions, delayed execution, shared-budget violations, and transaction ordering, are also relevant to conventional financial agents. Although the paper incorporates blockchain-specific primitives such as ERC-20 allowances, signed authorizations, and cross-domain execution, it does not clearly distinguish which safety challenges fundamentally depend on these primitives or what new evaluation capabilities their semantics necessitate.
- The vertical spacing above several section titles appears to have been improperly modified. Please restore the standard conference-template spacing to ensure formatting compliance.
7 comments
Thank you, reviewer Pefj! (2/5)
Specifically, we evaluate four defense configurations: (D0) No Defense, serving as the baseline; (D1) Prompt-Only Defense, which instructs agents to verify user constraints before taking financial actions; (D2) Request-Time Policy Enforcement, which checks proposed actions against user policies using the latest available observations; and (D3) State-Bound Policy Enforcement, which extends D2 by re-reading on-chain state and revalidating constraints at the execution boundary. These configurations allow us to distinguish the effects of prompt-level guidance, action-level policy enforcement, and execution-time state verification.
We construct a 50-instance defense evaluation suite. Each defense is evaluated under matched ordinary and adversarial conditions using DeepSeek V4.1 Flash, GPT-6 Sol, Qwen3.8 Flash, and Claude Opus 5.5, yielding 1,600 agent trajectories. We additionally evaluate 400 deterministic attacker episodes to validate defense behavior independently of LLM decision-making. We measure both Attack Success Rate (ASR), defined here as the adversarial policy-violation rate, and Safe Completion (J) on matched benign tasks.
Table R1: Comparison of defense effectiveness across four LLM agents (Attack ASR %, lower is better).
| Defense | Qwen3.8 Flash | DeepSeek V4.1 Flash | GPT-6 Sol | Claude Opus 5.5 | Fixture | Overall (Live) |
|---|---|---|---|---|---|---|
| D0: No Defense | 46 | 34 | 44 | 34 | 74 | 39.5 |
| D1: Prompt-Only | 36 | 22 | 28 | 30 | 74 | 29 |
| D2: Request-Time Enforcement | 32 | 28 | 34 | 24 | 68 | 29.5 |
| D3: State-Bound Enforcement | 18 | 14 | 14 | 22 | 36 | 17 |
As shown in Table R1, D3 consistently achieves the lowest attack ASR across all four LLM agents, reducing the overall violation rate from 39.5% without defense to 17.0%, a 22.5-percentage-point reduction. The reductions are observed across all four models. To examine the security–utility trade-off, we also evaluate safe task completion on matched benign controls.
Table R2: Benign-task safe completion (J %, higher is better).
| Defense | Qwen3.8 Flash | DeepSeek V4.1 Flash | GPT-6 Sol | Claude Opus 5.5 | Overall (Live) |
|---|---|---|---|---|---|
| D0: No Defense | 96 | 88 | 86 | 62 | 83 |
| D1: Prompt-Only | 90 | 68 | 74 | 74 | 76.5 |
| D2: Request-Time Enforcement | 98 | 92 | 78 | 58 | 81.5 |
| D3: State-Bound Enforcement | 98 | 90 | 74 | 58 | 80 |
As shown in Table R2, D3 maintains an overall benign safe-completion rate of 80.0%, compared with 83.0% without defense. This indicates that the substantial reduction in policy violations is accompanied by a 3.0-percentage-point decrease in benign task utility, although the impact varies across models. These results also demonstrate why defense effectiveness should be evaluated jointly with task completion, rather than solely through reduced violation rates.
Mechanism-Level Analysis. Beyond aggregate comparisons, we further investigate how different defenses mitigate specific execution-dependency failures. Request-time enforcement (D2) eliminates all observed violations involving unsafe borrowing requests and collateral-ratio constraints across all four models, demonstrating the effectiveness of action-level policy validation. Furthermore, state-bound enforcement (D3) reduces violations caused by state changes between observation and execution to only one out of five instances per model, highlighting the additional benefit of execution-time state revalidation. However, violations involving persistent delegated authority and delayed redemption remain, indicating that such risks require additional authority-lifecycle enforcement beyond state-bound checks. These findings clarify the distinct contributions and limitations of different defense mechanisms, providing actionable insights into the design of safer agent systems.
These additional experiments provide systematic empirical evidence of the effectiveness and limitations of different mitigation strategies within CryptoDojo, substantially strengthening the completeness and practical relevance of our defense evaluation. These results demonstrate that CryptoDojo not only systematically evaluates the effectiveness and utility trade-offs of different defense mechanisms, but also reveals substantial residual safety risks even under layered enforcement mechanisms. In particular, persistent delegated authority and delayed execution remain challenging, highlighting the need for more comprehensive lifecycle-aware defenses that enforce user constraints across authorization, execution, and post-session effects. These findings further underscore the value of CryptoDojo as a systematic testbed for developing and evaluating future agent safety mechanisms. We will incorporate these defense comparisons and mechanism-level findings into the revised manuscript to further demonstrate CryptoDojo's value as a framework for studying and improving agent safety.
Thank you, reviewer Pefj! (3/5)
Response to Questions
Unclear crypto-specific novelty. Many of the evaluated failure mechanisms, including stale-state decisions, delayed execution, shared-budget violations, and transaction ordering, are also relevant to conventional financial agents. Although the paper incorporates blockchain-specific primitives such as ERC-20 allowances, signed authorizations, and cross-domain execution, it does not clearly distinguish which safety challenges fundamentally depend on these primitives or what new evaluation capabilities their semantics necessitate.
We thank the reviewer for this insightful comment regarding the crypto-specific novelty of CryptoDojo. We agree that temporal-compositional failure mechanisms, such as stale-state decisions, delayed execution, and shared-budget violations, are not exclusive to crypto agents. However, our novelty lies not in claiming these general failure mechanisms are unique to crypto, but in capturing their distinct manifestations and policy consequences under blockchain-specific execution semantics. Unlike centrally coordinated execution workflows, blockchain transactions may be subject to externally determined inclusion and ordering, while finalized transactions generally cannot be unilaterally reversed. More specifically, ERC-20 allowances can establish persistent spending permissions [5], ERC-2612 permits separate the deadline for establishing an allowance from its subsequent lifetime [6], and ERC-3009 signed transfers introduce explicit validity windows and nonce-dependent authorization states [7]. Furthermore, cross-domain operations can separate authorization, submission, and final commitment across asynchronous execution environments. These characteristics necessitate evaluation capabilities that track authorization lifecycles, transaction ordering, commitment evidence, and downstream effects beyond agent termination, rather than evaluating isolated decisions or transaction outcomes alone.
To further clarify our crypto-specific contributions, we conduct a systematic comparison with representative prior work, including SmarTest (USENIX Security 2021) [1], FORAY (CCS 2024) [2], CrAIBench [3], and PACE [4]. As summarized in Table R3, these studies address complementary aspects of smart-contract vulnerabilities, DeFi attack synthesis, Web3 agent context manipulation, and transaction-level policy enforcement. In contrast, CryptoDojo provides an integrated evaluation framework combining controlled execution-time interventions, delegation-to-effect reconstruction, and joint evaluation of Task Completion (U), Policy Violation (V), and Safe Completion (J). Specifically, its Transaction-lifecycle Intervention Compiler controls when external events affect execution, while its Delegation-to-Effect Analyzer reconstructs authorization, transaction commitment, and downstream effects throughout the evaluation horizon, including after the agent terminates. T-CAM Bench further instantiates these capabilities through temporal-compositional attacks targeting state, authority, ordering, and shared-resource dependencies.
Table R3: Comparison with representative crypto-security and agent-evaluation studies.
| Work | ERC-20 Allowance Lifecycle | Signed Authorization Semantics | Cross-Domain Effects | Post-Session Authority Tracking | Joint U/V/J |
|---|---|---|---|---|---|
| SmarTest [1] | NR | NR | NR | NR | NR |
| FORAY [2] | NR | NR | NR | NR | NR |
| CrAIBench [3] | NR | NR | Cross-chain tasks | NR | NR |
| PACE [4] | Approval-related checks | Signed PDRs | NR | NR | NR |
| CryptoDojo (Ours) | Modeled | Modeled | Modeled | ✓ | ✓ |
Controlled Comparison with Conventional Financial Workflows. To further empirically substantiate the importance of crypto-specific evaluation capabilities, we conduct an additional controlled experiment comparing traditional financial and crypto-agent workflows under matched financial objectives and user constraints. Specifically, we construct 15 paired task instances across three scenarios: (1) Balance Drift, (2) Shared-Budget Constraints, and (3) Authorization and Delayed Execution, with five parameterizations per scenario. We evaluate the same DeepSeek agent under ordinary and adversarial conditions in both environments, yielding 60 valid execution episodes. The traditional environment employs a local payment-processor simulator based on the documented Stripe Treasury OutboundPayment workflow, while the crypto environment uses deployed ERC-20 and ERC-2612 functionality on a private Anvil EVM. Both environments preserve aligned financial objectives, constraints, and initial states while retaining their respective execution, cancellation, and authorization semantics. The results are summarized in Table R4.
Thank you, reviewer Pefj! (1/5)
Response to Weaknesses
We thank the reviewer for raising this important concern regarding the distinction between agent-level safety and execution-environment limitations. We agree that certain temporal-compositional failures cannot be reliably prevented by agent reasoning alone without sufficient state observability or execution-time enforcement. We would like to clarify that CryptoDojo evaluates the end-to-end safety of agent–tool–execution configurations, rather than attributing all observed policy violations solely to agent reasoning failures. Its evaluation separately measures Task Completion (U), Policy Violation (V), and Safe Completion (J), while explicitly accounting for the observation and enforcement capabilities of the execution environment.
To further substantiate this distinction, we conduct additional controlled experiments comparing four defense configurations: (D0) No Defense, (D1) Prompt-Only Guidance, (D2) Request-Time Policy Enforcement, and (D3) State-Bound Enforcement. Importantly, D2 and D3 employ identical policy rules, with D3 introducing only execution-time state re-observation and revalidation, enabling us to isolate the contribution of execution-level safeguards. We evaluate four LLM agents on 50 matched tasks under ordinary and adversarial conditions, yielding 1,600 valid execution episodes. The results are summarized in Table R0.
Table R0: Impact of execution-time safeguards on adversarial policy violations and benign task utility (%).
As shown in Table R0, execution-environment safeguards substantially influence observed safety outcomes. Overall ASR decreases from 39.5% without defense to 17.0% under D3. More importantly, adding execution-time state revalidation reduces ASR from 29.5% under D2 to 17.0% under D3, an additional 12.5-percentage-point reduction consistently observed across all four models, while benign safe completion remains comparable (81.5% vs. 80.0%). A deterministic attacker fixture further confirms the effect of execution-level safeguards, with ASR decreasing from 68% to 36% between D2 and D3.
Mechanism-level analysis further clarifies these distinctions. Request-time enforcement eliminates the observed unsafe borrowing violations involving collateral-ratio constraints, while execution-time revalidation reduces observation-to-execution state-drift violations to one out of five instances per model. Nevertheless, residual violations involving persistent delegated authority and delayed redemption remain, indicating that different failure mechanisms require protections at different execution boundaries.
These experiments provide empirical evidence that agent safety depends critically on tool observability and execution safeguards, beyond reasoning capabilities alone. The remaining 17.0% violation rate under the strongest evaluated defense further highlights the importance of lifecycle-aware protections. These findings strengthen CryptoDojo's value as a system-level safety evaluation framework for identifying environmental limitations and guiding the development of safer agent systems. We will incorporate these results and clarify the distinction between agent-level and execution-environment limitations in the revised manuscript.
We thank the reviewer for this valuable comment regarding the lack of systematic defense evaluations. We agree that evaluating explicit mitigation strategies is important for understanding not only the vulnerabilities identified by T-CAM, but also how different defense mechanisms address them. To directly address this concern, we conduct additional defense experiments comparing four progressively stronger mitigation configurations across four LLM agents, covering both attack resistance and benign task utility.