- Rating
- 4(+1)
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This article proposes and constructs a heterogeneous graph benchmark FinFraudBench for financial fraud detection, aiming to address the problem of existing graph benchmarks simplifying real scenarios into homographs or multi-relationship graphs in financial fraud detection, as well as the difficulty in fully reflecting real fraud detection scenarios. In this heterogeneous graph benchmark, the authors construct two large-scale heterograph datasets, CreditCard-Fraud and BankTrans-Fraud, using a unified schema containing 6 entity types and 14 relationship types, while providing reproducible evaluation protocols. The authors also classify existing baseline methods and compare these baselines on the constructed benchmark. Experimental results show that heterogeneous graph methods perform well, highlighting the importance of fully utilizing heterogeneous entities and their relationships in heterogeneous graphs.
Critical strengths
-
The paper builds a heterogeneous graph benchmark in the field of financial fraud detection. Existing benchmarks often simplify financial fraud graphs into homographs or multi-relational graphs, without fully considering some important features in financial fraud scenarios, such as severe label imbalance and limited annotated training samples.
-
The paper provides comprehensive baseline evaluation criteria and some valuable insights. It presents reproducible evaluation protocols and evaluates some existing baselines on the constructed benchmarks. It emphasizes the importance of considering heterogeneous graph structures in financial fraud detection.
Critical weaknesses
-
The paper was evaluated only on a balanced subset of mini-tests. In real scenarios, label distributions tend to be unbalanced, but the paper only shows evaluation results on a balanced subset of mini-tests.
-
The paper mainly considers a transductive setting. Under this settings, the training phase can access the context information of the cross-split. Considering that in some scenarios, the context information of the cross-split may not be visible, but the paper does not consider this situation.
Questions for authors
-
Can the authors supplement test results from different Baselines on full test data that retain the original unbalanced label distribution to better reflect model performance in real-world scenarios?
-
- Considering that cross-split contextual information may not be visible in some real-world scenarios, can the authors supplement the evaluation results in such cases?.
4 comments
Has our response resolved your concerns?
Dear Reviewer 1wpF,
We appreciate your insightful and helpful review! Your suggestions on evaluation at the original class distribution and generalization to unseen transactions helped us extend the evaluation. We added full-test results and an inductive evaluation with independently constructed, disconnected split graphs, and discuss what these results indicate about the available graph context.
If our response has resolved your concerns and clarified any ambiguities, we respectfully hope you might consider raising the (confidence) score. Should you have further questions or need additional clarification, we would be happy to discuss them. Thank you again for your time and effort in reviewing our manuscript. Your feedback has been instrumental in improving our research!
Best regards,
Authors
As the authors have addressed my concerns, I am happy to raise my scores to 4.
Thanks for your helpful feedback!
Dear Reviewer 1wpF,
Thank you for your insightful and constructive review. We sincerely appreciate your careful consideration of our responses and your decision to raise your scores to 4.
Best regards,
Authors
Response to Reviewer 1wpF
We thank Reviewer 1wpF for the positive assessment of the benchmark's motivation, scale, limited-label setting, and baseline coverage. We answer both questions first, then address the corresponding weaknesses.
We use balanced mini-tests to compare methods with equal fraud and benign representation in threshold-dependent metrics, while averaging over five sampled subsets reduces dependence on any single test draw. To assess performance under the original class imbalance, we also report all seven metrics on the complete test sets in Appendix C.2 (Table 8): 555,719 transactions (0.3860% fraud) for CreditCard-Fraud and 1,782,993 transactions (0.1495% fraud) for BankTrans-Fraud. The full-test results continue to show strong performance from heterogeneous and fraud-oriented methods, supporting the main family-level conclusion, although the leading method varies across datasets and metrics.
Appendix C.3 (Table 9) reports inductive full-test results using the same transaction splits as the main experiments. We construct the train, validation, and test graphs independently, compute auxiliary-node statistics within each split, and keep the split graphs disconnected during message passing. Thus, nodes, edges, and these statistics are not shared across splits; models are trained on labeled training transactions, selected on validation, and evaluated on the complete test split. In Table 9, GraphSAGE leads in Macro-F1 and Fraud F1 on CreditCard-Fraud, while ConsisGAD leads on both metrics on BankTrans-Fraud. The leading method thus depends on the dataset and metric. As discussed in Appendix C.3, disconnecting the split graphs limits cross-split entity context, which may reduce the benefit of heterogeneous neighborhood aggregation. These results motivate combining heterogeneous relational modeling with mechanisms tailored to fraud detection in inductive settings.
As detailed in our response to Q1, the revised manuscript now reports complete-test metrics at the source-split prevalence in Appendix C.2 (Table 8), alongside the balanced mini-test comparison.
As detailed in our response to Q2, the revised manuscript now includes an inductive evaluation with separately constructed and disconnected split graphs.
We have revised the manuscript in response to your suggestions. We greatly appreciate your helpful comments, as they will undoubtedly help us improve the quality of our article. If our response has successfully addressed your concerns and clarified the ambiguities, we respectfully hope that you consider raising the score. Should you have any further questions or require additional clarification, we would be delighted to engage in further discussion. Once again, we sincerely appreciate your time and effort in reviewing our manuscript. Your feedback has been invaluable in improving our research.