- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Bgent, a multi-task LLM agent for blockchain investigations. It uses LLMs to extract tool-operation principles and investigation procedures from public tutorials, synthesizing them into a reusable tool taxonomy and investigation skill. The agent conducts investigations using these resources and employs LLM-generated executable programs to verify factual claims against collected blockchain evidence.
Critical strengths
- The paper presents a unified LLM agent framework for multiple blockchain investigation tasks.
- Executable claim checking provides an interesting approach to improving the factual reliability of agent-generated explanations.
Critical weaknesses
- The proposed framework appears to merely integrate existing technologies, including LLM-based workflow extraction, tool-using agents, and executable factual verification. However, its methodological novelty beyond their integration remains unclear. For example, the authors should clarify how the proposed investigation workflow differs from common Search Agent or ReAct-style tool-use pipelines, how the skill construction differs from existing workflow learning approaches (e.g., Agent Workflow Memory), and what advantages the executable claim-checking mechanism offers over existing tool-assisted verification methods (e.g., FacTool and CRITIC). The authors need to clearly articulate where the innovation lies.
- The distillation pipeline primarily relies on GPT's existing search and summarization capabilities. The author need to clearly specify how sources are selected, how conflicting or potentially incorrect guidance is resolved, or how the generated procedures are validated. Moreover, the resulting skill appears to be constructed offline and reused statically at runtime. If the knowledge sources and resulting skill are fixed, the necessity of the proposed multi-stage LLM distillation pipeline remains unclear compared with manually constructed prompts or directly providing the relevant tutorials. If the framework is intended to support automated or dynamic knowledge acquisition, the authors should establish how newly discovered sources are evaluated, integrated, and verified. Comparisons with RAG and online search-based agents would help demonstrate the benefits of the proposed approach in terms of task performance, reliability, and efficiency.
- How does the method ensure that LLMs fully and correctly understand the original claims when generating verification code, without misinterpretation or omission? The paper lacks reliability evaluations, including the impact of different LLM capabilities and model sizes.
- The authors explicitly acknowledge differences in evaluation settings in Section 4.1 and Table 1. In particular, Table 1 states that the account-classification results use "different sampled addresses and a different metric protocol." Section 4.1 also reports 102 fault-localization cases for Bgent versus 115 for FaultSeeker. For attack detection, the paper compares locally sampled results with published baseline scores without establishing a same-instance evaluation.
Questions for authors
Please see the concerns and questions raised in the Critical Weaknesses section.