acceptodds
Desk rejected at ICLR 2027

CryptoDojo: A Framework for Safety and Security Evaluation of Crypto Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Crypto agents translate user instructions into transfers, spending permissions, and commitments that can persist long after the decisions that created them. An individually authorized action can become unsafe when state changes or other actions interact with it, and evaluating actions in isolation misses this. We introduce CryptoDojo, a framework that injects controlled interventions during execution and reconstructs their financial effects to grade both policy compliance and task completion, while agents, attacks, defenses, and tasks vary. On CryptoDojo, we propose the Temporal-Compositional Attack Model (T-CAM), which targets the state, authority, ordering, and shared-resource dependencies among authorized actions, and instantiate it as T-CAM Bench. Across four agent configurations on 200 paired instances, safe completion drops from 53.5–90.5% on ordinary tasks to 0.5–14.0% under attack. Adversarial violation rates differ widely (3.5–28.0%), but safe completion does not. In 138 of 145 violations, the requested operation settles in full, and the violation lies in its timing or state, such as payment after a deadline or after a revocation. Configurations that violate less also settle fewer operations, so their safe completion is no higher. Evaluations of crypto agents should measure settlement and policy compliance together and track authority exercised after the agent stops.

Primary area:
datasets and benchmarks
Keywords:
Crypto, LLM Agents, Safety, Security, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PefjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CryptoDojo, a framework for evaluating cryptocurrency agents under adversarial changes to transaction execution. Its central premise is that individually authorized actions can violate user policies due to changes in financial state, transaction ordering, persistent authority, or interactions across workflows. The authors propose the Temporal-Compositional Attack Model (T-CAM) and construct a benchmark of 200 paired ordinary/adversarial instances. Across four LLM-based agent configurations, safe completion drops from 53.5-90.5% under ordinary conditions to 0.5–14.0% under attack. The results also highlight that low policy-violation rates do not necessarily imply high task completion.

Critical strengths
  1. Execution-aware evaluation of financial safety and determinisitc criteria. CryptoDojo evaluates actual financial outcomes rather than relying solely on agents' decisions or explanations. Its separation of task completion, policy violation, and safe completion is particularly useful for identifying transactions that settle successfully but violate timing, authorization, or financial constraints.

  2. Paired evaluation reveals meaningful safety-utility tradeoffs. The ordinary/adversarial pairing controls for tasks and initial conditions, while the experimental results demonstrate substantial differences between agents' completion and violation rates.

Critical weaknesses
  1. Unclear whether the benchmark measures agent safety or limitations of the execution environment. Some attacks deliberately introduce state changes between an agent's observation and transaction commitment. Without mechanisms for observing such changes or enforcing constraints atomically, an agent cannot reliably prevent these violations, regardless of its reasoning capabilities.

  2. Lack of evaluated defenses or mitigation strategies. Although CryptoDojo is designed to support varying defenses, the experimental evaluation considers only four agent configurations without explicit mitigation mechanisms or systematic defense comparisons.

Questions for authors

See weaknesses, and:

  • Unclear crypto-specific novelty. Many of the evaluated failure mechanisms, including stale-state decisions, delayed execution, shared-budget violations, and transaction ordering, are also relevant to conventional financial agents. Although the paper incorporates blockchain-specific primitives such as ERC-20 allowances, signed authorizations, and cross-domain execution, it does not clearly distinguish which safety challenges fundamentally depend on these primitives or what new evaluation capabilities their semantics necessitate.
  • The vertical spacing above several section titles appears to have been improperly modified. Please restore the standard conference-template spacing to ensure formatting compliance.
7 comments
AuthorsOpenReview ↗

Thank you, reviewer Pefj! (1/5)

Response to Weaknesses

Unclear whether the benchmark measures agent safety or limitations of the execution environment. Some attacks deliberately introduce state changes between an agent's observation and transaction commitment. Without mechanisms for observing such changes or enforcing constraints atomically, an agent cannot reliably prevent these violations, regardless of its reasoning capabilities.

We thank the reviewer for raising this important concern regarding the distinction between agent-level safety and execution-environment limitations. We agree that certain temporal-compositional failures cannot be reliably prevented by agent reasoning alone without sufficient state observability or execution-time enforcement. We would like to clarify that CryptoDojo evaluates the end-to-end safety of agent–tool–execution configurations, rather than attributing all observed policy violations solely to agent reasoning failures. Its evaluation separately measures Task Completion (U), Policy Violation (V), and Safe Completion (J), while explicitly accounting for the observation and enforcement capabilities of the execution environment.

To further substantiate this distinction, we conduct additional controlled experiments comparing four defense configurations: (D0) No Defense, (D1) Prompt-Only Guidance, (D2) Request-Time Policy Enforcement, and (D3) State-Bound Enforcement. Importantly, D2 and D3 employ identical policy rules, with D3 introducing only execution-time state re-observation and revalidation, enabling us to isolate the contribution of execution-level safeguards. We evaluate four LLM agents on 50 matched tasks under ordinary and adversarial conditions, yielding 1,600 valid execution episodes. The results are summarized in Table R0.

Table R0: Impact of execution-time safeguards on adversarial policy violations and benign task utility (%).

DefenseQwen ASR ↓DeepSeek ASR ↓GPT-6 ASR ↓Opus ASR ↓Overall ASR ↓Benign J ↑
D0: No Defense4634443439.583
D1: Prompt-Only362228302976.5
D2: Request-Time Enforcement3228342429.581.5
D3: State-Bound Enforcement181414221780

As shown in Table R0, execution-environment safeguards substantially influence observed safety outcomes. Overall ASR decreases from 39.5% without defense to 17.0% under D3. More importantly, adding execution-time state revalidation reduces ASR from 29.5% under D2 to 17.0% under D3, an additional 12.5-percentage-point reduction consistently observed across all four models, while benign safe completion remains comparable (81.5% vs. 80.0%). A deterministic attacker fixture further confirms the effect of execution-level safeguards, with ASR decreasing from 68% to 36% between D2 and D3.

Mechanism-level analysis further clarifies these distinctions. Request-time enforcement eliminates the observed unsafe borrowing violations involving collateral-ratio constraints, while execution-time revalidation reduces observation-to-execution state-drift violations to one out of five instances per model. Nevertheless, residual violations involving persistent delegated authority and delayed redemption remain, indicating that different failure mechanisms require protections at different execution boundaries.

These experiments provide empirical evidence that agent safety depends critically on tool observability and execution safeguards, beyond reasoning capabilities alone. The remaining 17.0% violation rate under the strongest evaluated defense further highlights the importance of lifecycle-aware protections. These findings strengthen CryptoDojo's value as a system-level safety evaluation framework for identifying environmental limitations and guiding the development of safer agent systems. We will incorporate these results and clarify the distinction between agent-level and execution-environment limitations in the revised manuscript.

Lack of evaluated defenses or mitigation strategies. Although CryptoDojo is designed to support varying defenses, the experimental evaluation considers only four agent configurations without explicit mitigation mechanisms or systematic defense comparisons.

We thank the reviewer for this valuable comment regarding the lack of systematic defense evaluations. We agree that evaluating explicit mitigation strategies is important for understanding not only the vulnerabilities identified by T-CAM, but also how different defense mechanisms address them. To directly address this concern, we conduct additional defense experiments comparing four progressively stronger mitigation configurations across four LLM agents, covering both attack resistance and benign task utility.

AuthorsOpenReview ↗

Thank you, reviewer Pefj! (2/5)

Specifically, we evaluate four defense configurations: (D0) No Defense, serving as the baseline; (D1) Prompt-Only Defense, which instructs agents to verify user constraints before taking financial actions; (D2) Request-Time Policy Enforcement, which checks proposed actions against user policies using the latest available observations; and (D3) State-Bound Policy Enforcement, which extends D2 by re-reading on-chain state and revalidating constraints at the execution boundary. These configurations allow us to distinguish the effects of prompt-level guidance, action-level policy enforcement, and execution-time state verification.

We construct a 50-instance defense evaluation suite. Each defense is evaluated under matched ordinary and adversarial conditions using DeepSeek V4.1 Flash, GPT-6 Sol, Qwen3.8 Flash, and Claude Opus 5.5, yielding 1,600 agent trajectories. We additionally evaluate 400 deterministic attacker episodes to validate defense behavior independently of LLM decision-making. We measure both Attack Success Rate (ASR), defined here as the adversarial policy-violation rate, and Safe Completion (J) on matched benign tasks.

Table R1: Comparison of defense effectiveness across four LLM agents (Attack ASR %, lower is better).

DefenseQwen3.8 FlashDeepSeek V4.1 FlashGPT-6 SolClaude Opus 5.5FixtureOverall (Live)
D0: No Defense463444347439.5
D1: Prompt-Only362228307429
D2: Request-Time Enforcement322834246829.5
D3: State-Bound Enforcement181414223617

As shown in Table R1, D3 consistently achieves the lowest attack ASR across all four LLM agents, reducing the overall violation rate from 39.5% without defense to 17.0%, a 22.5-percentage-point reduction. The reductions are observed across all four models. To examine the security–utility trade-off, we also evaluate safe task completion on matched benign controls.

Table R2: Benign-task safe completion (J %, higher is better).

DefenseQwen3.8 FlashDeepSeek V4.1 FlashGPT-6 SolClaude Opus 5.5Overall (Live)
D0: No Defense9688866283
D1: Prompt-Only9068747476.5
D2: Request-Time Enforcement9892785881.5
D3: State-Bound Enforcement9890745880

As shown in Table R2, D3 maintains an overall benign safe-completion rate of 80.0%, compared with 83.0% without defense. This indicates that the substantial reduction in policy violations is accompanied by a 3.0-percentage-point decrease in benign task utility, although the impact varies across models. These results also demonstrate why defense effectiveness should be evaluated jointly with task completion, rather than solely through reduced violation rates.

Mechanism-Level Analysis. Beyond aggregate comparisons, we further investigate how different defenses mitigate specific execution-dependency failures. Request-time enforcement (D2) eliminates all observed violations involving unsafe borrowing requests and collateral-ratio constraints across all four models, demonstrating the effectiveness of action-level policy validation. Furthermore, state-bound enforcement (D3) reduces violations caused by state changes between observation and execution to only one out of five instances per model, highlighting the additional benefit of execution-time state revalidation. However, violations involving persistent delegated authority and delayed redemption remain, indicating that such risks require additional authority-lifecycle enforcement beyond state-bound checks. These findings clarify the distinct contributions and limitations of different defense mechanisms, providing actionable insights into the design of safer agent systems.

These additional experiments provide systematic empirical evidence of the effectiveness and limitations of different mitigation strategies within CryptoDojo, substantially strengthening the completeness and practical relevance of our defense evaluation. These results demonstrate that CryptoDojo not only systematically evaluates the effectiveness and utility trade-offs of different defense mechanisms, but also reveals substantial residual safety risks even under layered enforcement mechanisms. In particular, persistent delegated authority and delayed execution remain challenging, highlighting the need for more comprehensive lifecycle-aware defenses that enforce user constraints across authorization, execution, and post-session effects. These findings further underscore the value of CryptoDojo as a systematic testbed for developing and evaluating future agent safety mechanisms. We will incorporate these defense comparisons and mechanism-level findings into the revised manuscript to further demonstrate CryptoDojo's value as a framework for studying and improving agent safety.

AuthorsOpenReview ↗

Thank you, reviewer Pefj! (3/5)

Response to Questions

Unclear crypto-specific novelty. Many of the evaluated failure mechanisms, including stale-state decisions, delayed execution, shared-budget violations, and transaction ordering, are also relevant to conventional financial agents. Although the paper incorporates blockchain-specific primitives such as ERC-20 allowances, signed authorizations, and cross-domain execution, it does not clearly distinguish which safety challenges fundamentally depend on these primitives or what new evaluation capabilities their semantics necessitate.

We thank the reviewer for this insightful comment regarding the crypto-specific novelty of CryptoDojo. We agree that temporal-compositional failure mechanisms, such as stale-state decisions, delayed execution, and shared-budget violations, are not exclusive to crypto agents. However, our novelty lies not in claiming these general failure mechanisms are unique to crypto, but in capturing their distinct manifestations and policy consequences under blockchain-specific execution semantics. Unlike centrally coordinated execution workflows, blockchain transactions may be subject to externally determined inclusion and ordering, while finalized transactions generally cannot be unilaterally reversed. More specifically, ERC-20 allowances can establish persistent spending permissions [5], ERC-2612 permits separate the deadline for establishing an allowance from its subsequent lifetime [6], and ERC-3009 signed transfers introduce explicit validity windows and nonce-dependent authorization states [7]. Furthermore, cross-domain operations can separate authorization, submission, and final commitment across asynchronous execution environments. These characteristics necessitate evaluation capabilities that track authorization lifecycles, transaction ordering, commitment evidence, and downstream effects beyond agent termination, rather than evaluating isolated decisions or transaction outcomes alone.

To further clarify our crypto-specific contributions, we conduct a systematic comparison with representative prior work, including SmarTest (USENIX Security 2021) [1], FORAY (CCS 2024) [2], CrAIBench [3], and PACE [4]. As summarized in Table R3, these studies address complementary aspects of smart-contract vulnerabilities, DeFi attack synthesis, Web3 agent context manipulation, and transaction-level policy enforcement. In contrast, CryptoDojo provides an integrated evaluation framework combining controlled execution-time interventions, delegation-to-effect reconstruction, and joint evaluation of Task Completion (U), Policy Violation (V), and Safe Completion (J). Specifically, its Transaction-lifecycle Intervention Compiler controls when external events affect execution, while its Delegation-to-Effect Analyzer reconstructs authorization, transaction commitment, and downstream effects throughout the evaluation horizon, including after the agent terminates. T-CAM Bench further instantiates these capabilities through temporal-compositional attacks targeting state, authority, ordering, and shared-resource dependencies.

Table R3: Comparison with representative crypto-security and agent-evaluation studies.

WorkERC-20 Allowance LifecycleSigned Authorization SemanticsCross-Domain EffectsPost-Session Authority TrackingJoint U/V/J
SmarTest [1]NRNRNRNRNR
FORAY [2]NRNRNRNRNR
CrAIBench [3]NRNRCross-chain tasksNRNR
PACE [4]Approval-related checksSigned PDRsNRNRNR
CryptoDojo (Ours)ModeledModeledModeled✓✓

Controlled Comparison with Conventional Financial Workflows. To further empirically substantiate the importance of crypto-specific evaluation capabilities, we conduct an additional controlled experiment comparing traditional financial and crypto-agent workflows under matched financial objectives and user constraints. Specifically, we construct 15 paired task instances across three scenarios: (1) Balance Drift, (2) Shared-Budget Constraints, and (3) Authorization and Delayed Execution, with five parameterizations per scenario. We evaluate the same DeepSeek agent under ordinary and adversarial conditions in both environments, yielding 60 valid execution episodes. The traditional environment employs a local payment-processor simulator based on the documented Stripe Treasury OutboundPayment workflow, while the crypto environment uses deployed ERC-20 and ERC-2612 functionality on a private Anvil EVM. Both environments preserve aligned financial objectives, constraints, and initial states while retaining their respective execution, cancellation, and authorization semantics. The results are summarized in Table R4.

Reviewer_JzZSOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CryptoDojo, a framework for evaluating crypto agents under changes in execution state, authorization, transaction ordering, and shared resources. It proposes T-CAM, an attack model targeting dependencies between locally authorized actions and their eventual effects. Experiments across four agent configurations and 200 paired instances reveal substantial declines in safe task completion under attack.

Critical strengths
  1. Evaluation beyond individual decisions. The framework tracks execution effects, including authority exercised after agent termination, capturing risks that isolated action checks can miss.

  2. Joint measurement of safety and utility. Separating settlement, policy violations, and safe completion reveals that successful transactions may violate user constraints, while low violation rates may coexist with frequent task non-completion.

Critical weaknesses
  1. Limited domain coverage. Whether an agent can achieve its objective while respecting user constraints after the conditions it relies on change during execution is a common challenge across domains. However, this paper examines the issue exclusively in the financial domain, which limits the generalizability of its findings. Consequently, its relevance to the design and deployment of most current agent systems is limited.

  2. *Limited evaluation. Although the paper reports aggregate outcomes and failure profiles across execution-dependency categories, it does not systematically establish whether tasks remain safely completable under the adversarial interventions. This makes it difficult to distinguish appropriate stopping from failure to find a feasible, policy-compliant execution. Furthermore, the evaluation provides limited causal diagnosis of failures: it does not adequately disentangle failures arising from the agents’ action choices from those caused by limitations in the tool interfaces or execution safeguards. These analysis would provide more insights in building agents' safety ability.

Questions for authors

N/A

6 comments
AuthorsOpenReview ↗

Thank you, reviewer JzZS! (1/3)

Response to Weaknesses

Limited domain coverage. Whether an agent can achieve its objective while respecting user constraints after the conditions it relies on change during execution is a common challenge across domains. However, this paper examines the issue exclusively in the financial domain, which limits the generalizability of its findings. Consequently, its relevance to the design and deployment of most current agent systems is limited.

We thank the reviewer for raising this important concern regarding the domain coverage and generalizability of CryptoDojo. We agree that maintaining user constraints under changing execution conditions is a general challenge across agentic systems. However, we would like to clarify an important distinction between the domain-specific scope of T-CAM Bench and the broader applicability of CryptoDojo as an evaluation framework. While T-CAM Bench provides a high-fidelity instantiation of temporal-compositional safety failures in crypto agents, CryptoDojo is designed as an extensible evaluation infrastructure that separates agent execution, environment interaction, intervention scheduling, and outcome assessment through a modular experiment contract. Its core design principles—controlled execution, structured evidence collection, and independent outcome evaluation—are applicable to broader agentic workflows through domain-specific adapters. Therefore, the crypto-specific scope of our current safety benchmark should not be conflated with the applicability of the underlying evaluation framework.

To directly substantiate this distinction with empirical evidence, we extend CryptoDojo to two independently developed, non-financial benchmarks: (1) Cloud-OpsBench [9], covering cloud infrastructure fault diagnosis, and (2) τ²-bench Airline [10], covering airline customer-service workflows. Specifically, we evaluate 58 Cloud-OpsBench cases spanning all 57 fault categories and all 50 base tasks from τ²-bench Airline. Importantly, the integration preserves the benchmarks' original task definitions, tool interfaces, execution semantics, and native scoring procedures. Only the necessary model invocation, orchestration, and evidence-recording adapters are introduced, without replacing the official evaluators. We evaluate four LLM configurations, and the results are summarized in Table 1.

Table 1: Cross-domain benchmark integration and native evaluation fidelity.

ModelCloud-OpsBench JRAτ²-bench Airline RewardNative Scoring Replay
DeepSeek V4.1 Flash53/58 (91.4%)43/50 (86.0%)108/108
GPT-6 Sol49/58 (84.5%)39/50 (78.0%)108/108
Claude Opus 5.556/58 (96.6%)42/50 (84.0%)108/108
Qwen3.8 Flash50/58 (86.2%)43/49 (87.8%)107/107
Overall208/232 (89.7%)167/199 (83.9%)431/431 (100%)

As shown in Table 1, CryptoDojo successfully supports two distinct non-financial agent workflows while preserving their original evaluation semantics. Across four models and 432 planned executions, 431 produced scoreable outcomes, and all 431 results matched independent replay using the official benchmark evaluators. Crucially, this is not merely an architectural claim of extensibility: these experiments provide direct empirical validation that CryptoDojo can orchestrate heterogeneous agent workflows, preserve domain-specific tool and task semantics, and reproduce native evaluation outcomes without altering the underlying benchmarks. This demonstrates that the framework's applicability is not restricted to crypto-financial environments.

Furthermore, our focus on crypto agents is a deliberate choice motivated by distinctive safety requirements, rather than an inherent limitation of the broader evaluation methodology. Crypto agents operate in environments involving potentially irreversible transactions, persistent delegated authority, and asynchronous execution across multiple actors, making execution-aware safety evaluation particularly important. The growing deployment of crypto agents, exemplified by Coinbase's Agentic Wallets [1] and Olas's reported 11.7 million agent-to-agent transactions [2], further motivates dedicated evaluation infrastructure. Importantly, domain-specific evaluation is a well-established research paradigm, as demonstrated by SWE-bench (ICLR 2024) [3], LegalAgentBench (ACL 2025) [4], MedAgentBench (NEJM AI 2025) [5], FinBen (NeurIPS 2024) [6], TravelPlanner (ICML 2024) [7], and CyberGym-E2E (ICML 2026) [8]. These studies illustrate how focusing on specialized domains enables realistic modeling of workflows, constraints, and verifiable outcomes that are difficult to capture in general-purpose evaluations. CryptoDojo follows this established direction while providing an extensible framework that can accommodate other agent environments.

AuthorsOpenReview ↗

Thank you, reviewer JzZS! (2/3)

These additional experiments provide empirical evidence of CryptoDojo's broader applicability beyond financial agents, helping to address the reviewer's concern regarding limited domain coverage. We distinguish this validated framework extensibility from the domain-specific adversarial findings of T-CAM Bench, whose measured safety outcomes remain specific to crypto agents. Together, the cross-domain integration results, preserved native evaluation fidelity, and established value of domain-specific benchmarks substantiate CryptoDojo's broader methodological relevance without overstating the scope of its current safety findings. We will incorporate these results and clarify the distinction between framework-level generalizability and benchmark-specific empirical conclusions in the revised manuscript.

[1] Erik Reppel and Josh Nickerson. "Introducing Agentic Wallets: Give Your Agents the Power of Autonomy." Coinbase Developer Platform, 2026. https://www.coinbase.com/developer-platform/discover/launches/agentic-wallets

[2] Olas. "Q1 2026 Roundup." Olas Blog, 2026. https://olas.network/blog/olas-q1-2026-roundup

[3] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. "SWE-bench: Can Language Models Resolve Real-world GitHub Issues?" In International Conference on Learning Representations (ICLR), 2024. https://proceedings.iclr.cc/paper_files/paper/2024/hash/edac78c3e300629acfe6cbe9ca88fb84-Abstract-Conference.html

[4] Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, Wuyue Wang, Yiqun Liu, and Minlie Huang. "LegalAgentBench: Evaluating LLM Agents in Legal Domain." In Annual Meeting of the Association for Computational Linguistics (ACL), 2025. https://aclanthology.org/2025.acl-long.116/

[5] Yixing Jiang, Kameron C. Black, Gloria Geng, Danny Park, James Zou, Andrew Y. Ng, and Jonathan H. Chen. "MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents." NEJM AI, 2(9), AIdbp2500144, 2025. https://doi.org/10.1056/AIdbp2500144

[6] Qianqian Xie, Weiguang Han, Zhengyu Chen, et al. "FinBen: A Holistic Financial Benchmark for Large Language Models." In Advances in Neural Information Processing Systems (NeurIPS), 2024. https://proceedings.nips.cc/paper_files/paper/2024/hash/adb1d9fa8be4576d28703b396b82ba1b-Abstract-Datasets_and_Benchmarks_Track.html

[7] Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. "TravelPlanner: A Benchmark for Real-World Planning with Language Agents." In International Conference on Machine Learning (ICML), 2024. https://proceedings.mlr.press/v235/xie24j.html

[8] Tianneng Shi, Robin Rheem, Dongwei Jiang, Mona Wang, Francisco De La Riega, Zhun Wang, Jingzhi Jiang, Alexander Cheung, Sean Tai, Jonah Cha, Jianhong Tu, Gabriel Han, Chenguang Wang, Jingxuan He, Wenbo Guo, and Dawn Song. "CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities." In International Conference on Machine Learning (ICML), 2026. https://proceedings.mlr.press/v306/shi26r.html

[9] Yilun Wang, Guangba Yu, Haiyu Huang, Yujie Huang, Zirui Wang, Pengfei Chen, and Michael R. Lyu. "Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems." arXiv preprint arXiv:2603.00468, 2026. https://arxiv.org/abs/2603.00468

[10] Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik R. Narasimhan. "τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment." In International Conference on Machine Learning (ICML), 2026. https://proceedings.mlr.press/v306/barres26a.html

Limited evaluation. Although the paper reports aggregate outcomes and failure profiles across execution-dependency categories, it does not systematically establish whether tasks remain safely completable under the adversarial interventions. This makes it difficult to distinguish appropriate stopping from failure to find a feasible, policy-compliant execution. Furthermore, the evaluation provides limited causal diagnosis of failures: it does not adequately disentangle failures arising from the agents’ action choices from those caused by limitations in the tool interfaces or execution safeguards. These analysis would provide more insights in building agents' safety ability.

We thank the reviewer for highlighting the need to establish safe-completion feasibility and better diagnose failures. We supplement our evaluation with a secondary analysis of archived reference executions, transaction receipts, and corresponding model outcomes.

(1) Safe-completion feasibility. We document 28 reference executions achieving U = 1, V = 0, J = 1, using the exposed agent tools and validated through the original evaluator and recorded execution trajectory.

AuthorsOpenReview ↗

Thank you, reviewer JzZS! (3/3)

Table 2: Safe-reference evidence under the registered adversarial scheduling rules.

Reference mechanismInstancesVerified behavior
Existing recovery operation5Authorization paths close; subsequent payments revert.
Explicit nonce cancellation and epoch revocation21Outstanding authorizations close; subsequent redemptions revert.
Conditional-trigger avoidance2Safe completion occurs without triggering the state mutation.

The references demonstrate safe completion using available recovery capabilities, with subsequent external execution attempts explicitly recorded and rejected. We report the remaining two separately, because avoiding a conditional trigger does not establish feasibility after its state mutation. Crucially, initial-state feasibility differs from feasibility at stopping. In one collateral-constrained borrowing task, the fixed borrowing satisfies the constraint initially, but the documented post-mutation state yields a collateral ratio of 138.5%, below the required 150%, with no exposed repayment or collateral-restoration operation. This illustrates why stopping may preserve safety even when completion was initially possible. Only three of the 572 nonviolating incompletions overlap with our reference set, all on this same task, whose reference avoids the state-change trigger. We therefore do not classify these executions as avoidable failures based on the reference alone.

These findings motivate distinguishing recovery availability, successful authorization closure, and state-dependent completion feasibility. They provide additional evidence addressing the reviewer’s concerns without claiming an exhaustive causal attribution of all failures. We will incorporate this analysis and clarify that U/V/J measure outcomes, while feasibility and execution evidence are needed to interpret those outcomes.

Reviewer_m5d5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce CryptoDojo, a new eval framework for measuring policy compliance and task completion of crypto agents under controlled interventions. They also introduce T-CAM, a new attack model, and instantiate it on CryptoDojo as T-CAM Bench. The central takeaway is that related benchmarks need to measure policy compliance and task completion jointly, rather than separately.

Critical strengths

Both CryptoDojo and T-CAM Bench are timely and useful contributions to the part of the community interested in crypto agents. The empirical work clearly motivates the stated takeaways, namely that task completion and policy compliance have to be considered jointly. While this joint view is just a particular instance of the security-utility trade-off, it is useful and actionable advice for practitioners in this particular subarea.

Critical weaknesses

I have not been able to identify a critical weakness.

Questions for authors
  • how would the authors further increase the realism of CryptoDojo wrt real-world crypto agents applications?
  • how would the authors extend the existing benchmark to multi-agent systems of interacting crypto agents, and how could T-CAM bench be extended to this setting?
4 comments
AuthorsOpenReview ↗

Thank you, reviewer m5d5! (1/2)

how would the authors further increase the realism of CryptoDojo wrt real-world crypto agents applications

We thank the reviewer for this constructive question regarding the realism of CryptoDojo in real-world crypto-agent applications. We agree that increasing realism requires not only representative financial tasks, but also faithful modeling of end-to-end execution lifecycles, including wallet authorization, transaction submission, commitment, and downstream effects. CryptoDojo already provides a foundation through stateful execution, controlled interventions, transaction-level evidence reconstruction, and post-session authority tracking. Furthermore, our supplementary experiments incorporate deployed ERC-20 and ERC-2612 functionality on a private Anvil EVM, demonstrating the feasibility of evaluating concrete on-chain execution semantics within our framework.

Building on this foundation, we identify three concrete directions for improving realism. First, application-driven workflows: existing infrastructure such as Coinbase's Agentic Wallets [1] already supports autonomous payments, trading, and programmable spending limits, motivating realistic end-to-end agent tasks. We would incorporate DEX portfolio rebalancing with execution-price and slippage constraints grounded in Uniswap's trading semantics [2], lending position management with collateral and liquidation-risk constraints based on Aave [3], and delegated payments with explicit authorization scopes and expiration conditions. Second, production-faithful execution: we would extend our integration to deployed protocol contracts, standard wallet-signing interfaces, and version-pinned blockchain forks, preserving realistic transaction states while retaining controlled execution and reproducibility. Foundry's Anvil supports block-specific chain forking and configurable mining behavior [6], providing a practical foundation for this extension. Third, realistic adversarial conditions: we would incorporate historically grounded liquidity changes, transaction-ordering variations, and authorization-lifecycle risks. Prior empirical research has documented real-world vulnerabilities associated with persistent token approvals [4], while ERC-2612 explicitly defines signed authorization deadlines and nonce semantics [5], providing concrete foundations for constructing realistic security scenarios.

These extensions would preserve CryptoDojo's core evaluation rigor. Each scenario would retain matched ordinary and adversarial conditions, independently verified safe reference executions, and trace-based assessment of Task Completion (U), Policy Violation (V), and Safe Completion (J). By combining realistic protocol semantics with controlled interventions and reproducible execution evidence, CryptoDojo can progressively bridge benchmark-based safety evaluation and deployment-relevant crypto-agent workflows without sacrificing experimental interpretability and reliability.


References

[1] Erik Reppel and Josh Nickerson. "Introducing Agentic Wallets: Give Your Agents the Power of Autonomy." Coinbase Developer Platform, 2026. https://www.coinbase.com/developer-platform/discover/launches/agentic-wallets

[2] Uniswap Labs. "Single Swaps." Uniswap Developer Documentation. https://developers.uniswap.org/docs/protocols/v3/guides/swapping/single-hop-swapping

[3] Aave. "Health Factor & Liquidations." Aave Documentation. https://www.aave.com/help/borrowing/liquidations

[4] Nicola Ruaro, Fabio Gritti, Dongyu Meng, Robert McLaughlin, Ilya Grishchenko, Christopher Kruegel, and Giovanni Vigna. "Approve Once, Regret Forever: On the Exploitation of Ethereum's Approve-TransferFrom Ecosystem." In 34th USENIX Security Symposium, 2025. https://www.usenix.org/conference/usenixsecurity25/presentation/ruaro

[5] Martin Lundfall. "ERC-2612: Permit Extension for EIP-20 Signed Approvals." Ethereum Improvement Proposals, 2020. https://eips.ethereum.org/EIPS/eip-2612

[6] Foundry. "Anvil: Ethereum Development Framework." Official Documentation. https://www.getfoundry.sh/anvil/index.html

AuthorsOpenReview ↗

Thank you, reviewer m5d5! (2/2)

how would the authors extend the existing benchmark to multi-agent systems of interacting crypto agents, and how could T-CAM bench be extended to this setting?

We thank the reviewer for this insightful question regarding the extension of CryptoDojo and T-CAM Bench to multi-agent crypto systems. We believe this is a natural extension of our framework, as CryptoDojo already models shared-resource dependencies, delegated authority, asynchronous execution, and interactions across multiple sessions and external actors. Building on its modular experiment contract and execution-event scheduling, we envision extending the current session-composition setting to explicit communication, coordination, and delegation among autonomous agents, while preserving the framework's controlled execution and trace-based evaluation principles.

First, extending CryptoDojo to multi-agent workflows. We would introduce a multi-agent orchestration layer in which each agent maintains its own identity, permissions, observations, and execution history. For example, in a DeFi portfolio-rebalancing workflow, a Planning Agent proposes transactions, a Risk Agent validates collateral and spending constraints, and an Execution Agent submits transactions using delegated wallet authority. Agent interactions would be represented through structured messages and authorization records, drawing on emerging standards such as the Agent2Agent (A2A) Protocol [1] and Agent Payments Protocol (AP2) [2]. Crucially, CryptoDojo's existing execution and evidence-reconstruction architecture could be extended to link agent messages, delegated permissions, transaction submissions, and realized on-chain effects across agent boundaries.

Second, extending T-CAM Bench to multi-agent temporal-compositional attacks. The extended benchmark would systematically evaluate (1) stale-state propagation, where an execution agent acts on information approved by another agent before an intervening state change; (2) cross-agent authority persistence, where delegated permissions remain exercisable after revocation or task termination; and (3) fragmented constraint enforcement, where individually authorized actions by different agents collectively violate shared spending or exposure limits. The intervention scheduler would control message delays, ordering, state changes, and authorization updates at inter-agent handoff boundaries. Following the established multi-agent evaluation paradigm exemplified by MultiAgentBench (ACL 2025) [3], we would additionally examine how different coordination structures affect these risks.

Importantly, the extension would preserve CryptoDojo's core evaluation methodology. Each scenario would retain matched ordinary and adversarial conditions, explicitly bounded attacker capabilities, and trace-based evaluation of Task Completion (U), Policy Violation (V), and Safe Completion (J) at the system level. The extended evidence records would further identify the agent actions, communication events, and authorization dependencies involved in observed outcomes. This would enable CryptoDojo to move beyond single-agent and session-level safety evaluation toward systematic analysis of emergent safety risks arising from inter-agent coordination and delegated execution, while retaining the reproducibility and execution-level accountability central to our framework.


References

[1] Agent2Agent Project. "Agent2Agent (A2A) Protocol Specification." Official Protocol Documentation, 2026. https://a2a-protocol.org/dev/specification/

[2] Google and AP2 Contributors. "Agent Authorization Framework." Agent Payments Protocol (AP2) Documentation, 2025. https://ap2-protocol.org/ap2/agent_authorization/

[3] Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji, and Jiaxuan You. "MultiAgentBench: Evaluating the Collaboration and Competition of LLM Agents." In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025, pp. 8580–8622. https://aclanthology.org/2025.acl-long.421/

Thanks for your response. My rating remains unchanged. If accepted, I ask that the failure analysis be reconciled with the adversarial split and also please clarify for which tasks safe completion remains at all feasible after the intervention.

Thank you, reviewer m5d5! Both points will be addressed in the camera-ready version, and the key numbers are summarized below.

First, reconciling the failure analysis with the adversarial split. On the original split used for the main results, the 717 adversarial episodes without safe completion consist of 145 policy violations and 572 non-violating incompletions. Violations count as unsafe outcomes regardless of feasibility. Of the 572 incompletions, only 3 occur on the 28 tasks with a verified safe reference, all on the same task whose reference avoids the trigger, as reported to Reviewer JzZS. The remaining 569 occur on tasks without a verified safe reference. To diagnose failures where a safe path exists, the failure analysis reported to Reviewer JzZS uses the revised 200-task split described below, with four agent configurations under matched ordinary and adversarial conditions (1,600 episodes, 800 adversarial). All 393 categorized failures come from adversarial episodes, and every adversarial policy violation (385) is categorized. Ordinary-condition failures are reported separately:

CategoryAdv. violationAdv. incompletionOrd. violationOrd. incompletion
Incorrect execution ordering349000
Authorization-lifecycle errors32038
Guard bypass4000
Agent action/tool errors08010
Agent-attributed subtotal3858318
Task-side issues (not attributed)03200
Total38540318

The 393 categorized failures (385 + 8) are the agent-attributed adversarial failures. The remaining 32 adversarial episodes involve no violation and trace to the task rather than the agent: 31 occur on 21 FX tasks whose conditional instruction can be read as conflicting with the added recovery authorization, and 1 occurs on a task whose reference avoids the trigger. We are revising these tasks and will report the rerun. 380 of the 393 attributed failures (96.7%) occur on tasks that the same agent safely completed under the ordinary condition.

Second, which tasks remain safely completable after the intervention.

Status after intervention (original split)Tasks
Verified safe reference that handles the state change26
Verified safe reference that avoids the conditional trigger2
Completable only by skipping a required check42
Only partial safe completion found13
No complete safe execution found (bounded search)117

Post-intervention feasibility is therefore established for 26 tasks. For the other 174, we do not claim infeasibility, since the search is bounded and not a proof. We only claim that non-completion on these tasks cannot be attributed to the agent without per-task evidence. The 0.5% to 14.0% adversarial safe-completion range should therefore be read against the 28 tasks with a verified reference, rather than as a pure capability measure.

The revised split addresses this by adding authorized recovery, fee-settlement, and authorization-closure obligations to the same 200 tasks. All 400 reference executions (200 tasks × 2 conditions) achieve J = 1 with the same tools, permissions, intervention schedule, and execution budgets as the evaluated agents, and all 200 negative controls violate the policy. For 198 tasks, the reference handles every intervention. The remaining 2 are the trigger-avoiding tasks above: their references handle the added fee intervention but still avoid the original trigger, and we are revising them. On this split, adversarial safe completion ranges from 20.0% to 70.5% across the four agents, and in 343 of the 349 ordering failures, the agent had already observed every compliant quote, yet executed the operation before settling any fee.

In the camera-ready version, we will (i) include the full accounting above and state which split each result uses, (ii) add a per-task appendix giving each task's feasibility status on both splits, the safe strategy used by the reference, and the expected safe behavior when completion is not established, and (iii) report results after revising the 23 tasks noted above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.