acceptodds
Withdrawn from ICLR 2027

Bgent: A Multi-Task Blockchain Investigation Agent with Executable Claim Checks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Blockchain investigations connect transaction execution, account behavior, and contract state to explain on-chain activity. Task-specific prediction systems assist investigators, while evidence collection and interpretation still require substantial expertise. We present Bgent, a multi-task blockchain investigation agent that combines tutorial-derived guidance with executable checks of factual claims. Offline distillation extracts tool operations and investigation procedures, producing a tool taxonomy and a reusable skill. At runtime, the investigator retrieves evidence and submits an answer with explicit factual claims. A planner selects checkable claims, and independent validators execute Python predicates over saved tool responses. Failed checks trigger revision within the same investigation session. Experiments on Ethereum cover account classification, attack-transaction detection, and smart-contract fault localization. Bgent achieves 96.45% mean F1 on valid attack-detection decisions and 56.86% Recall@1 in fault localization. Removing distilled guidance reduces account-classification F1 from 80.40% to 38.14%. Executable checking corrects specific factual errors in explanations while leaving the reported classification scores unchanged.

Primary area:
interpretability and explainable AI
Keywords:
Agent, Blockchain

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1vHgOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Bgent, a multi-task LLM agent for blockchain investigations. It uses LLMs to extract tool-operation principles and investigation procedures from public tutorials, synthesizing them into a reusable tool taxonomy and investigation skill. The agent conducts investigations using these resources and employs LLM-generated executable programs to verify factual claims against collected blockchain evidence.

Critical strengths
  1. The paper presents a unified LLM agent framework for multiple blockchain investigation tasks.
  2. Executable claim checking provides an interesting approach to improving the factual reliability of agent-generated explanations.
Critical weaknesses
  1. The proposed framework appears to merely integrate existing technologies, including LLM-based workflow extraction, tool-using agents, and executable factual verification. However, its methodological novelty beyond their integration remains unclear. For example, the authors should clarify how the proposed investigation workflow differs from common Search Agent or ReAct-style tool-use pipelines, how the skill construction differs from existing workflow learning approaches (e.g., Agent Workflow Memory), and what advantages the executable claim-checking mechanism offers over existing tool-assisted verification methods (e.g., FacTool and CRITIC). The authors need to clearly articulate where the innovation lies.
  2. The distillation pipeline primarily relies on GPT's existing search and summarization capabilities. The author need to clearly specify how sources are selected, how conflicting or potentially incorrect guidance is resolved, or how the generated procedures are validated. Moreover, the resulting skill appears to be constructed offline and reused statically at runtime. If the knowledge sources and resulting skill are fixed, the necessity of the proposed multi-stage LLM distillation pipeline remains unclear compared with manually constructed prompts or directly providing the relevant tutorials. If the framework is intended to support automated or dynamic knowledge acquisition, the authors should establish how newly discovered sources are evaluated, integrated, and verified. Comparisons with RAG and online search-based agents would help demonstrate the benefits of the proposed approach in terms of task performance, reliability, and efficiency.
  3. How does the method ensure that LLMs fully and correctly understand the original claims when generating verification code, without misinterpretation or omission? The paper lacks reliability evaluations, including the impact of different LLM capabilities and model sizes.
  4. The authors explicitly acknowledge differences in evaluation settings in Section 4.1 and Table 1. In particular, Table 1 states that the account-classification results use "different sampled addresses and a different metric protocol." Section 4.1 also reports 102 fault-localization cases for Bgent versus 115 for FaultSeeker. For attack detection, the paper compares locally sampled results with published baseline scores without establishing a same-instance evaluation.
Questions for authors

Please see the concerns and questions raised in the Critical Weaknesses section.

Reviewer_C47AOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Bgent, a blockchain investigation agent combining tutorial-derived procedural guidance with executable checks of factual claims. Public tutorials are distilled into a tool taxonomy and a reusable investigation skill. At runtime, an investigator collects on-chain evidence and submits an answer with explicit claims; separate validators generate Python programs to check selected claims against saved tool responses, with failures triggering revision. The system is evaluated on scenarios of Ethereum account classification, attack-transaction detection, and smart-contract fault localization.

Critical strengths

S1: The paper addresses a meaningful distinction between producing a correct task label and providing a factually accurate explanation. Its examples show that an answer can retain the same label while containing incorrect amounts, dates, or counterparty labels.

S2: The evaluation covers three task types, and the account-classification ablation shows a substantial benefit from the combined tutorial-derived guidance. Reporting failed and invalid outputs separately from valid-answer metrics is also helpful.

Critical weaknesses

W1: Limited novelty. The paper does not sufficiently establish what methodological advance this combination offers over existing tool-using and verification-based agents. The distillation ablation removes both the tool taxonomy and the investigation skill, so it cannot distinguish the benefit of the proposed extraction-and-synthesis procedure from that of simply providing better domain instructions.

W2: The paper does not touch the central reasoning challenges of blockchain investigation. The method primarily provides a retrospective checking over existing investigation but has limited evidence of the ability to construct a mechanism from a fresh attack, connect activity across transactions and contracts, or produce a complete, causally justified investigation report. Autonomous discovery and real-time monitoring need not be requirements for this paper.

W3: Removing code checking leaves the reported account-classification metrics unchanged. This does not rule out a benefit to explanation quality, but that benefit is supported mainly by four correction examples. There is no systematic measurement of claim-selection coverage, validator accuracy, false acceptance or rejection, or factual-error reduction across the three tasks. With at most three submitted claims per round, it is also unclear how much of an explanation is actually verified. The central checking component therefore lacks a sufficiently rigorous evaluation.

W4: Data sources are under-specified. The paper describes adapter operations but does not identify their underlying data provider or explain whether they use a self-hosted node, an explorer API, or another indexing service. It does not report block-height ranges, a fixed retrieval snapshot, the number of distinct token contracts, or the total transaction evidence retrieved during investigation. Account-transfer queries are capped at 10,000 records per category, without a clear account of truncation handling. In addition, the use of present-day histories and the latest contract state makes the evidence boundary important: the results support retrospective analysis but do not establish performance using only information available at the incident time. These omissions hinder reproducibility and assessment of evidence completeness.

Questions for authors

Please address the points that directly correspond to the concerns raised in the Weaknesses section. No new experiments are required. I would consider a higher rating if the authors could clarify these.

Reviewer_YfJfOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents Bgent, a multi-task blockchain investigation agent that supports three tasks: account classification, attack-transaction detection, and smart-contract fault localization. The proposed framework consists of two main components: (1) Tutorial Distillation, which extracts tool-use procedures and investigation strategies from publicly available blockchain investigation tutorials and synthesizes them into reusable skills; and (2) Executable Claim Checking, which uses LLM-generated Python scripts to verify factual claims against collected on-chain evidence and provides execution feedback to correct potential errors.

The authors evaluate Bgent on three Ethereum investigation tasks, reporting 80.40% F1 for account classification, 96.45% mean F1 for attack detection (on valid outputs), and 56.86% Recall@1 for fault localization. The ablation study shows that tutorial distillation substantially improves account-classification performance, while executable claim checking does not improve classification accuracy but helps correct several factual errors in the generated explanations.

Critical strengths

The paper presents a unified blockchain investigation framework that can handle multiple tasks, including account classification, attack detection, and fault localization. This unified design is a useful step toward more general-purpose blockchain investigation agents. Additionally, by retaining collected evidence and executable validation scripts, the framework improves the transparency and auditability of the investigation process. The demonstrated ability to correct factual errors in generated explanations also offers practical value.

Critical weaknesses
  • Limited reliability guarantees for executable claim checking.

One of the main contributions of this paper is the use of executable programs to verify LLM-generated factual claims. However, these verification programs are themselves generated by LLMs, and their correctness is not independently guaranteed. While executing Python scripts against collected evidence provides an additional layer of verification, the approach may still inherit reliability issues from the LLM-generated checking logic. In particular, an incorrectly generated validator could potentially accept a false claim or reject a correct one.

More importantly, the paper does not provide a systematic evaluation of the correctness and coverage of the generated validators. For example, it would be helpful to understand how frequently the validators produce correct verdicts and what proportion of factual claims can be meaningfully verified. Without such an evaluation, it is difficult to assess the reliability and effectiveness of executable claim checking as a core technical contribution.

  • Insufficient experimental comparisons to establish the effectiveness of the proposed framework.

First, the paper compares its results with previously published results from other methods. However, these comparisons involve differences in evaluation datasets, sampling strategies, and evaluation protocols. Although the paper acknowledges some of these differences, the reported numbers do not constitute direct apples-to-apples comparisons. As a result, it is difficult to draw strong conclusions about the relative performance of Bgent against existing approaches.

Second, the experimental evaluation would benefit from a stronger general-purpose agent baseline. The current implementation uses DeepSeek-v4.1-Flash as the investigator backbone, but does not compare against a simpler tool-using agent built on a stronger foundation model, such as GPT-6 Astra, under the same evaluation conditions. This comparison seems particularly relevant given that the authors already employ GPT-6 Astra during the offline tutorial distillation stage. Without such a baseline, it remains unclear whether the additional complexity of the proposed framework provides substantial benefits over directly using a stronger LLM with access to the same investigation tools.

Questions for authors

Please refer to the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.