acceptodds
Under review as a conference paper at ICLR 2027

FinFraudBench: A Heterogeneous Graph Benchmark for Financial Fraud Detection

Abstract

The increasing complexity of digital financial systems has reshaped financial fraud detection from isolated transaction classification into relational risk reasoning over interconnected financial entities. This shift has motivated graph-based fraud detection, where models identify fraudulent nodes by exploiting dependencies among customers, cards, merchants, categories, and locations. However, despite rapid progress in graph-based methods, existing public benchmarks remain misaligned with real-world financial systems in two important aspects. First, they often simplify financial ecosystems into homogeneous or single-node-type multi-relational graphs, failing to preserve the multi-entity and multi-relational nature of financial data. Second, they rarely provide large-scale heterogeneous financial graph datasets with realistic operating conditions such as extreme class imbalance and limited label availability, making it difficult to assess the practical effectiveness of current methods. To address these gaps, we present **FinFraudBench**, a heterogeneous graph benchmark for financial fraud detection. FinFraudBench contains two heterogeneous graph datasets (**CreditCard-Fraud** and **BankTrans-Fraud**) with up to 8.99M nodes and 89.23M directed typed edges. Each dataset preserves six financial entity types, fourteen directed edge types, and natural fraud rates that mirror deployment constraints. With these datasets, we establish a standardized evaluation protocol covering both ranking and imbalance-sensitive classification metrics, and evaluate representative baselines. Extensive experiments yield empirical insights into current methods' limitations and suggest promising avenues for future research. FinFraudBench is available at https://anonymous.4open.science/r/FinFraudBench-B002.

Primary area:
datasets and benchmarks
Keywords:
financial fraud detection, heterogeneous graphs, graph neural networks, benchmark datasets

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_1wpFOpenReview ↗
Rating
4(+1)
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This article proposes and constructs a heterogeneous graph benchmark FinFraudBench for financial fraud detection, aiming to address the problem of existing graph benchmarks simplifying real scenarios into homographs or multi-relationship graphs in financial fraud detection, as well as the difficulty in fully reflecting real fraud detection scenarios. In this heterogeneous graph benchmark, the authors construct two large-scale heterograph datasets, CreditCard-Fraud and BankTrans-Fraud, using a unified schema containing 6 entity types and 14 relationship types, while providing reproducible evaluation protocols. The authors also classify existing baseline methods and compare these baselines on the constructed benchmark. Experimental results show that heterogeneous graph methods perform well, highlighting the importance of fully utilizing heterogeneous entities and their relationships in heterogeneous graphs.

Critical strengths
  1. The paper builds a heterogeneous graph benchmark in the field of financial fraud detection. Existing benchmarks often simplify financial fraud graphs into homographs or multi-relational graphs, without fully considering some important features in financial fraud scenarios, such as severe label imbalance and limited annotated training samples.

  2. The paper provides comprehensive baseline evaluation criteria and some valuable insights. It presents reproducible evaluation protocols and evaluates some existing baselines on the constructed benchmarks. It emphasizes the importance of considering heterogeneous graph structures in financial fraud detection.

Critical weaknesses
  1. The paper was evaluated only on a balanced subset of mini-tests. In real scenarios, label distributions tend to be unbalanced, but the paper only shows evaluation results on a balanced subset of mini-tests.

  2. The paper mainly considers a transductive setting. Under this settings, the training phase can access the context information of the cross-split. Considering that in some scenarios, the context information of the cross-split may not be visible, but the paper does not consider this situation.

Questions for authors
  1. Can the authors supplement test results from different Baselines on full test data that retain the original unbalanced label distribution to better reflect model performance in real-world scenarios?

    1. Considering that cross-split contextual information may not be visible in some real-world scenarios, can the authors supplement the evaluation results in such cases?.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 1wpF

We thank Reviewer 1wpF for the positive assessment of the benchmark's motivation, scale, limited-label setting, and baseline coverage. We answer both questions first, then address the corresponding weaknesses.

Q1: “Could you provide additional results on the original full test sets to demonstrate performance under naturally imbalanced fraud distributions?”

We use balanced mini-tests to compare methods with equal fraud and benign representation in threshold-dependent metrics, while averaging over five sampled subsets reduces dependence on any single test draw. To assess performance under the original class imbalance, we also report all seven metrics on the complete test sets in Appendix C.2 (Table 8): 555,719 transactions (0.3860% fraud) for CreditCard-Fraud and 1,782,993 transactions (0.1495% fraud) for BankTrans-Fraud. The full-test results continue to show strong performance from heterogeneous and fraud-oriented methods, supporting the main family-level conclusion, although the leading method varies across datasets and metrics.

Q2: “Could you include an inductive evaluation where new transactions are not accessible during training?”

Appendix C.3 (Table 9) reports inductive full-test results using the same transaction splits as the main experiments. We construct the train, validation, and test graphs independently, compute auxiliary-node statistics within each split, and keep the split graphs disconnected during message passing. Thus, nodes, edges, and these statistics are not shared across splits; models are trained on labeled training transactions, selected on validation, and evaluated on the complete test split. In Table 9, GraphSAGE leads in Macro-F1 and Fraud F1 on CreditCard-Fraud, while ConsisGAD leads on both metrics on BankTrans-Fraud. The leading method thus depends on the dataset and metric. As discussed in Appendix C.3, disconnecting the split graphs limits cross-split entity context, which may reduce the benefit of heterogeneous neighborhood aggregation. These results motivate combining heterogeneous relational modeling with mechanisms tailored to fraud detection in inductive settings.

W1: “The balanced mini-test results may not reflect the original class distribution.”

As detailed in our response to Q1, the revised manuscript now reports complete-test metrics at the source-split prevalence in Appendix C.2 (Table 8), alongside the balanced mini-test comparison.

W2: “The transductive setting limits assessment of generalization to unseen transactions.”

As detailed in our response to Q2, the revised manuscript now includes an inductive evaluation with separately constructed and disconnected split graphs.


We have revised the manuscript in response to your suggestions. We greatly appreciate your helpful comments, as they will undoubtedly help us improve the quality of our article. If our response has successfully addressed your concerns and clarified the ambiguities, we respectfully hope that you consider raising the score. Should you have any further questions or require additional clarification, we would be delighted to engage in further discussion. Once again, we sincerely appreciate your time and effort in reviewing our manuscript. Your feedback has been invaluable in improving our research.

AuthorsOpenReview ↗

Has our response resolved your concerns?

Dear Reviewer 1wpF,

We appreciate your insightful and helpful review! Your suggestions on evaluation at the original class distribution and generalization to unseen transactions helped us extend the evaluation. We added full-test results and an inductive evaluation with independently constructed, disconnected split graphs, and discuss what these results indicate about the available graph context.

If our response has resolved your concerns and clarified any ambiguities, we respectfully hope you might consider raising the (confidence) score. Should you have further questions or need additional clarification, we would be happy to discuss them. Thank you again for your time and effort in reviewing our manuscript. Your feedback has been instrumental in improving our research!

Best regards,
Authors

As the authors have addressed my concerns, I am happy to raise my scores to 4.

AuthorsOpenReview ↗

Thanks for your helpful feedback!

Dear Reviewer 1wpF,

Thank you for your insightful and constructive review. We sincerely appreciate your careful consideration of our responses and your decision to raise your scores to 4.

Best regards,
Authors

Reviewer_n34ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FinFraudBench, a heterogeneous graph benchmark for financial fraud detection constructed from two publicly available transaction-level datasets. The benchmark represents six types of financial entities and fourteen directed relation types, with up to 8.99 million nodes and 89.23 million edges. It provides standardized data splits and evaluation protocols under limited-label and highly imbalanced settings. Experiments compare representative non-GNN, homogeneous GNN, multi-relational, fraud-oriented, and heterogeneous GNN methods. The results suggest that heterogeneous GNNs perform best overall, highlighting the potential value of preserving entity types and relation semantics for financial fraud detection.

Critical strengths
  • Large-scale benchmark construction. The paper provides two substantial graph datasets that preserve multiple entity types and relation semantics, offering a potentially useful resource for research on heterogeneous financial fraud detection.

  • Systematic evaluation protocol. The benchmark includes labeled and unlabeled training data, multiple model families, and complementary evaluation metrics. This provides a useful foundation for controlled comparisons under limited-label and class-imbalanced settings.

Critical weaknesses
  • Insufficient novelty beyond heterogeneous graph representation. Heterogeneous graph learning has already been extensively explored in fraud detection, including GEM [1], MIE-HetGRL [2]. Therefore, preserving heterogeneous entities and relations is not, by itself, a sufficiently distinctive contribution. The benchmark's value would be more convincingly established by demonstrating advantages in other dimensions, such as more reliable and independently validated labels, richer and better-documented features, or more realistic representations of financial relationships. However, these advantages are not sufficiently established in the current manuscript.

  • Limited dataset diversity and scale coverage. Despite the large number of nodes and edges, the benchmark comprises only two datasets derived from existing tabular sources, limiting its coverage of different financial fraud scenarios and data characteristics. A more comprehensive benchmark could provide systematically constructed datasets at multiple scales (e.g., small, medium, and large), using consistent sampling procedures that preserve key structural and class-distribution properties. This would enable more informative evaluations of scalability, robustness, and performance under varying data availability.

  • Limited novelty and distinctiveness of the key findings. The two main findings, that heterogeneous GNNs perform best overall and that relational information is important for fraud detection, substantially overlap, as both primarily support the general benefit of exploiting heterogeneous relational structure. These findings offer limited additional insight beyond the expected advantages of graph-based modeling. More compelling findings would identify which entity types, relation types, or data characteristics contribute to performance, and how their effects vary across dataset scales and label-availability settings. Without such analyses, the empirical contribution remains relatively limited.

[1] Liu Z, Chen C, Yang X, et al. Heterogeneous graph neural networks for malicious account detection[C]//Proceedings of the 27th ACM international conference on information and knowledge management. 2018: 2077-2085. [2] Wu B, Chao K M, Li Y. Heterogeneous graph neural networks for fraud detection and explanation in supply chain finance[J]. Information Systems, 2024, 121: 102335.

Questions for authors

Check weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer n34Z

We thank Reviewer n34Z for recognizing the benchmark's scale and limited-label protocol. We address each weakness below.

W1: “Heterogeneous graph learning has already been explored in fraud detection; preserving heterogeneous entities and relations alone is not a sufficiently distinctive contribution.”

We agree that preserving heterogeneity alone is not the contribution. The two cited studies address different data and application settings: GEM [1] studies malicious-account detection using proprietary Alipay data that are not publicly available as a reusable benchmark, while MIE-HetGRL [2] focuses on supply-chain finance rather than routine-payment transactions. Our contribution is a public benchmark for routine-payment fraud that derives typed entities and semantically distinct relations, with a schema informed by risk-control practice at Alipay. It provides a unified six-node-type/fourteen-relation schema, documented construction, large-scale graphs, limited-label training, and full-test evaluation under severe class imbalance. The revised Introduction and Appendix A.2 clarify that the benchmark targets this specific scenario;

W2: “The benchmark contains only two datasets and lacks systematic small/medium/large scale coverage.”

We agree that the current release provides large-scale graphs rather than a systematically constructed small/medium/large suite. As reported in Appendix A.4, transaction nodes substantially outnumber auxiliary entity nodes in both datasets:

DatasetTransaction nodesAuxiliary entity nodesTotal nodes
CreditCard-Fraud1.85M2.76K1.86M
BankTrans-Fraud8.91M72.14K8.99M

Because transaction nodes substantially outnumber auxiliary entity nodes, benchmark scale can be controlled by sampling transactions while retaining the auxiliary entities and their semantic relation channels. With sampling that preserves the fraud prevalence and relation coverage, this allows smaller graph variants to retain the benchmark's heterogeneous structure.

W3: “The findings offer limited insight without identifying which entity types, relations, or data characteristics contribute to performance.”

Although the strong performance of heterogeneous and fraud-oriented methods is an intuitive pattern, our results provide empirical support for it. To add more specific insight, we added node-type ablations in Section 6.4 (Figure 5) of the revised manuscript. For each setting, we remove one node type and all its incident relations from the train, validation, and test graphs, then evaluate Macro-F1 and Fraud-F1 on the complete test set. The results show that the influential entity types vary across datasets and models: removing category nodes causes the largest HGT drop on CreditCard-Fraud, while merchant nodes have the largest effect on BankTrans-Fraud; for SeHGNN, customer removal has the largest effect on CreditCard-Fraud, and merchant removal again has the largest effect on BankTrans-Fraud. This analysis identifies dataset-dependent entity contributions.

[1] Liu, Z., Chen, C., Yang, X., et al. “Heterogeneous Graph Neural Networks for Malicious Account Detection.” Proceedings of the 27th ACM International Conference on Information and Knowledge Management, 2018, pp. 2077–2085.

[2] Wu, B., Chao, K. M., and Li, Y. “Heterogeneous Graph Neural Networks for Fraud Detection and Explanation in Supply Chain Finance.” Information Systems, vol. 121, 2024, article 102335.


We have revised the manuscript in response to your suggestions. We greatly appreciate your helpful comments, as they will undoubtedly help us improve the quality of our article. If our response has successfully addressed your concerns and clarified the ambiguities, we respectfully hope that you consider raising the score. Should you have any further questions or require additional clarification, we would be delighted to engage in further discussion. Once again, we sincerely appreciate your time and effort in reviewing our manuscript. Your feedback has been invaluable in improving our research.

AuthorsOpenReview ↗

Has our response resolved your concerns?

Dear Reviewer n34Z,

We appreciate your insightful and helpful review! Your comments on the benchmark's distinctiveness and the need for more specific empirical insights were particularly helpful. We clarified the routine-payment focus and added node-type ablations to examine how entity contributions vary across datasets.

If our response has resolved your concerns and clarified any ambiguities, we respectfully hope you might consider raising the score. Should you have further questions or need additional clarification, we would be happy to discuss them. Thank you again for your time and effort in reviewing our manuscript. Your feedback has been instrumental in improving our research!

Best regards,
Authors

I appreciate the authors’ targeted and constructive responses. However, I believe my third major concern has not yet been adequately addressed. Specifically, the main findings of the manuscript still appear to overlap with the general advantages of leveraging heterogeneous relational structures, without sufficiently highlighting interesting, task-specific insights from the datasets. The additional node-type ablation experiments also mainly show that the effects of removing different node types vary across datasets, rather than providing deeper insights into the underlying task. I believe the core findings of this study need to be further explored to deliver more meaningful and insightful conclusions. Nevertheless, I acknowledge the considerable strength of the study in terms of its large-scale datasets. Overall, I cautiously maintain my original score.

Dear Reviewer n34Z,

Thank you for your thoughtful follow-up and for recognizing the strength of our large-scale datasets. We sincerely appreciate your careful assessment and constructive feedback.

Best regards,
Authors

Reviewer_zRBSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem that most existing financial fraud detection benchmarks are homogeneous or multi-relational graphs, so they cannot capture the various entities and relations involved in financial transactions. To address this, the authors build two datasets by converting existing public transaction tables into heterogeneous graphs with multiple entity types such as customer, card, and merchant, and their relations. They also propose an evaluation protocol that reflects label scarcity, and evaluate baselines from various model families, from non-GNN models to heterogeneous GNNs. Based on the results, they claim that heterogeneous graph models show good overall performance.

Critical strengths
  1. Most existing benchmarks are homogeneous or multi-relational graphs, so they cannot capture the various entities and relations involved in financial transactions. The motivation is reasonable, and it is meaningful that the paper builds a financial fraud benchmark in the form of a heterogeneous graph that keeps multiple entity types and relations.

  2. Reflecting the difficulty of obtaining labels in practice, the labeled training data is limited to about 2% of all transactions, and unlabeled transactions are kept in the graph as context. This is a realistic setting that fits the label scarcity in actual fraud detection.

  3. This paper includes a wide range of baselines, from non-GNN models to homogeneous GNNs, multi-relational GNNs, fraud-oriented models, and heterogeneous GNNs. This allows a broad comparison of performance across different model designs.

Critical weaknesses
  1. The claim about the realism of the proposed benchmark is not well supported. The source data of CreditCard-Fraud, “Credit Card Transactions Fraud Detection Dataset” (Shenoy, 2020), is a simulated dataset. It is hard to say that a benchmark built on fraud rates and patterns generated by a simulator reflects real-world conditions. However, the paper does not clearly state this, and still uses expressions such as “natural fraud rate” and “realistic deployment”, which can be misleading. The authors should clearly describe the nature of the source data and adjust their claims about realism accordingly.

  2. The experimental results for the class imbalance motivation are insufficient. The paper uses extreme class imbalance as its motivation, but the experiments only report results on the balanced mini-test, where fraud and non-fraud have the same ratio. Even though the authors already built a test set that keeps the natural ratio (full test split), they do not report results on it. To be consistent with the motivation, the authors should report results on the natural-ratio test set, and analyze whether the performance ranking among models and the paper’s conclusions still hold under this condition.

  3. It is hard to trust the model comparison based on threshold-dependent metrics. There is no description of how the decision threshold of each model was set. In Appendix Tables 5 and 6, the Fraud Precision of most models is almost the same, while only the Fraud Recall differs a lot. This means the differences in Accuracy, Macro-F1, Fraud F1, and Fraud Recall mainly reflect differences in thresholds rather than model performance. So, for example, the conclusion that HGT is the best on these metrics may simply be a result of the threshold setting.

  4. The evidence for the conclusion that the graph structure is the source of the performance gain is not valid. According to Table 3, nodes such as card, customer, and merchant include features like transaction counts and amount statistics. Graph-based models can use this information, but the MLP baseline only uses the features of transaction nodes. Therefore, the performance gap between the two mixes the effect of the graph structure with the effect of the additional features. Also, boosting models such as XGBoost, which are widely used as baselines in financial fraud detection, are not included. The authors should compare with an MLP or a boosting model that also uses the features of the other nodes.

  5. The information needed for reproduction is insufficient. Since this paper is about a benchmark, reproducibility is very important. More details are needed on how the graph is converted into homogeneous or multi-relational graphs, how the hyperparameters of the baselines were searched, and how the scores were obtained from LLM prompting. Also, since the paper puts weight on the scalability of the benchmark, it should also report resource costs of the baselines, such as training time and memory usage.

Questions for authors
  1. In Table 3, nodes such as card, customer, and merchant include features like transaction counts and amount statistics. Over which period of transactions were these features computed? If the test period was also included, temporal leakage could occur, where future information is used in training even without label leakage.

  2. The test data of CreditCard-Fraud has the same size as the test file of the source data, and its fraud ratio is different from the training data. On the other hand, all splits of BankTrans-Fraud have the same fraud ratio, so it seems to be split randomly. Why are the two datasets split in different ways?

  3. The standard deviations in the paper seem to be computed over the five balanced mini-tests. Does the performance ranking among models still hold when the models are trained multiple times with different random seeds? The gaps between the top models are often small, so the conclusions of the paper could change depending on the seed results. I am also wondering whether the five mini-tests share the same fraud data.

14 comments
AuthorsOpenReview ↗

Response to Reviewer zRBS (1/2)

We thank Reviewer zRBS for the careful review. We answer the three questions first, followed by each of the five weaknesses.

Q1: “In Table 3, nodes such as card, customer, and merchant include features like transaction counts and amount statistics. Over which period of transactions were these features computed? If the test period was also included, temporal leakage could occur, where future information is used in training even without label leakage.”

These statistics are computed over the combined transaction records without a temporal cutoff, so they can include test transactions and transactions later than a target transaction. Fraud labels are excluded from their construction. In our current transductive setting, all splits share the complete graph and its non-label information, and transaction time is used as a feature rather than as a constraint on graph visibility or aggregate computation. Future work can adapt the benchmark to temporal evaluation by constructing time-based splits and recomputing graph context and entity statistics using only information available at each prediction time. We clarify the current construction in Appendix A.3.

Q2: “The test data of CreditCard-Fraud has the same size as the test file of the source data, and its fraud ratio is different from the training data. On the other hand, all splits of BankTrans-Fraud have the same fraud ratio, so it seems to be split randomly. Why are the two datasets split in different ways?”

CreditCard-Fraud retains the source-provided train/test partition. BankTrans-Fraud has no corresponding fixed train/test partition in the source, so we create a stratified random 80/20 split, which preserves the overall fraud prevalence across the splits. the difference follows the split information available in each source.

Q3: “Do the model rankings remain consistent across random seeds? Do the five mini-tests share fraud examples?”

The standard deviations over the five balanced mini-tests reflect test-subset variation, not variation across training seeds. The five subsets are randomly sampled without a disjointness constraint, so they may share fraud examples. Given the rebuttal-stage deadline, we ran three random seeds for four representative methods on one fixed mini-test subset to provide a seed-level reference. The table below reports mean ±\pm standard deviation for Macro-F1, Fraud-F1, AUROC, and AUPRC.

DatasetMethodFamilyMacro-F1Fraud-F1AUROCAUPRC
CreditCard-FraudGCNHomogeneous GNN0.4175±0.00310.4175 \pm 0.00310.1503±0.00550.1503 \pm 0.00550.8251±0.00430.8251 \pm 0.00430.8620±0.00290.8620 \pm 0.0029
CreditCard-FraudR-GCNMulti-relation0.6183±0.01950.6183 \pm 0.01950.4899±0.03160.4899 \pm 0.03160.8825±0.00580.8825 \pm 0.00580.9092±0.00270.9092 \pm 0.0027
CreditCard-FraudPMPFraud-oriented0.6953±0.01490.6953 \pm 0.01490.6118±0.02280.6118 \pm 0.02280.8583±0.00610.8583 \pm 0.00610.8949±0.00500.8949 \pm 0.0050
CreditCard-FraudHGTHeterogeneous0.7908±0.00670.7908 \pm 0.00670.7502±0.00940.7502 \pm 0.00940.8954±0.01800.8954 \pm 0.01800.9206±0.00880.9206 \pm 0.0088
BankTrans-FraudGCNHomogeneous GNN0.6616±0.00990.6616 \pm 0.00990.5623±0.01560.5623 \pm 0.01560.9276±0.00310.9276 \pm 0.00310.9285±0.00140.9285 \pm 0.0014
BankTrans-FraudR-GCNMulti-relation0.8421±0.02290.8421 \pm 0.02290.8175±0.02950.8175 \pm 0.02950.9578±0.00560.9578 \pm 0.00560.9695±0.00390.9695 \pm 0.0039
BankTrans-FraudPMPFraud-oriented0.8560±0.00180.8560 \pm 0.00180.8358±0.00220.8358 \pm 0.00220.9550±0.00440.9550 \pm 0.00440.9663±0.00360.9663 \pm 0.0036
BankTrans-FraudHGTHeterogeneous0.8569±0.00770.8569 \pm 0.00770.8371±0.01000.8371 \pm 0.01000.9753±0.00270.9753 \pm 0.00270.9805±0.00170.9805 \pm 0.0017

W1: “The source data are simulated, so the benchmark's realism is not well supported; expressions such as ‘natural fraud rate’ and ‘realistic deployment’ may be misleading.”

Although our source data are synthetic and cannot fully reproduce real-world transactions, public transaction-level records from routine-payment platforms are rarely released because of privacy and security concerns. We therefore construct the benchmark from public synthetic sources and use Alipay, a mobile-payment platform serving over one billion users, as a risk-control design reference. This informs a routine-payment schema with six node types and fourteen directed relation types, as well as graphs of up to 8.99M nodes and 89.23M edges. The protocol also reflects practical conditions through 2% labeled training transactions and highly imbalanced full-test sets, with fraud rates of 0.3860% and 0.1495%. Together, these design choices make the benchmark more representative of routine-payment fraud settings while preserving reproducibility.

AuthorsOpenReview ↗

Response to Reviewer zRBS (2/2)

W2: “The class-imbalance motivation is not supported by balanced mini-tests alone; analyze whether model rankings and conclusions hold on the full test sets.”

We use balanced mini-tests to give fraud and benign transactions equal weight in threshold-dependent metrics and average results over five subsets to reduce dependence on a single draw. To assess performance at the original class imbalance, Appendix C.2 (Table 8) reports all seven metrics on the complete test sets at their source-split fraud prevalence. Heterogeneous and fraud-oriented methods remain among the strongest under this setting, although the leading method varies by dataset and metric.

W3: “The decision thresholds are unspecified, so differences in classification metrics may reflect threshold choices rather than model performance.”

For each baseline, we select the classification threshold on the validation set by maximizing Fraud F1, then hold it fixed for test evaluation. We have added this rule to Section 5.2 of the revised manuscript. We also report AUROC and AUPRC, which are threshold-independent ranking metrics.

W4: “The graph and MLP inputs are mismatched, and widely used boosting baselines such as XGBoost are missing.”

We agree that comparing graph models only with a target-feature MLP would confound graph modeling with access to auxiliary-entity information. We therefore added a full-test comparison in Section 6.3 (Figure 4). MLP, XGBoost, LightGBM, and CatBoost use rows from the original source transaction tables augmented with associated-entity attributes and label-independent entity summaries, but process each row independently without graph edges or message passing. At source-split fraud prevalence, HAN achieves 0.7823/0.5660/0.7364 in Macro-F1/Fraud-F1/precision on CreditCard-Fraud, versus the strongest tabular scores of 0.7683/0.5385/0.4992; on BankTrans-Fraud, SeHGNN achieves 0.8937/0.7877/0.8459, versus 0.8063/0.6132/0.7396. The results show that enriching transaction rows alone does not match the performance of these heterogeneous graph models, supporting the value of modeling typed relations explicitly.

W5: “Reproduction details are insufficient for graph projections, baseline hyperparameters, LLM scoring, and resource costs.”

Appendix A.3 now details the graph projections. For the homogeneous GNN baselines, we treat all nodes as a single node type and collapse all edge relations into one adjacency structure. The multi-relational view uses transaction nodes with five separate shared-entity relation channels and samples neighbors on demand. Appendix B.1 (Table 6) reports the principal settings used for each baseline, including batch sizes, neighborhood sampling, and model hyperparameters; checkpoints are selected by validation AUPRC, and thresholds by validation Fraud F1 (Section 5.2). Appendix B.2 describes the Qwen2-7B-Instruct input construction, the zero- and four-shot prompts, excluded fields, and scoring Yes/No by normalized sequence log-probabilities. We also used method-specific batch sizes and sampling configurations to accommodate computational requirements at this graph scale.


We have revised the manuscript in response to your suggestions. We greatly appreciate your helpful comments, as they will undoubtedly help us improve the quality of our article. If our response has successfully addressed your concerns and clarified the ambiguities, we respectfully hope that you consider raising the score. Should you have any further questions or require additional clarification, we would be delighted to engage in further discussion. Once again, we sincerely appreciate your time and effort in reviewing our manuscript. Your feedback has been invaluable in improving our research.

Q1.

Thank you for the detailed responses to each point of my review. After reading the responses and the revised paper, there are a few things I would still like to check.

  • In Table 9, the performance of the heterogeneous models drops a lot compared to Table 8, and they are not the best in terms of Macro-F1 and Fraud F1 either. Could you explain how this result affects the conclusions of the paper?
  • In Table 9, GraphSAGE has the highest Macro-F1 (0.7085) and Fraud F1 (0.4197) on CreditCard-Fraud, but the table marks GAAP (0.6375) and PMP (0.2423) as the best, and Appendix C.3 also says that fraud-oriented methods are the highest. Could you check this?
Reviewer_Eoi4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a heterogeneous graph benchmark for financial fraud detection, named FinFraudBench. The authors argue that the current existing benchmarks for financial fraud detection often simplify the financial ecosystem into the homogeneous graphs and lack of large scale datasets that reflect the realistic conditions. The proposed benchmark addresses these issues by two large datasets, CreditCard-Fraud and BankTrans-Fraud. Each of them contains 6 financial entity types, 14 directed edge types and natural fraud rates. The authors also introduce a evaluation protocal and then evaluate the representative baselines, and provide the current method’s limitations and future research directions.

Critical strengths

The problem is well motivated and important. I think the benchmark and the analysis could pave the way for future work on financial fraud detection. I also appreciate that the authors provide the documentation and source code for the benchmark, which supports the reproducibility.

Critical weaknesses

Overall, I have two main concerns.

  • First, the details of the dataset construction is insufficiently described. The paper explains only briefly how the data was preprocessed, and created. Although Figure 2 gives an overview of the construction pipeline, key details are missing, such as how each component was implemented and how the graph structure (nodes and edges) was built. Because the datasets are one of the core contributions of the paper, a more thorough description of the pipeline would strengthen the work.

  • Second, the paper does not provide implementation details for the evaluated baselines. Information such as model configurations, hyperparameter search ranges, and training settings is needed for readers to judge whether the comparison is fair and to reproduce the results. Including these details would make the empirical findings more convincing.

Questions for authors
  • Could you include a table comparing the proposed benchmark with existing financial fraud detection benchmarks?

  • Could you provide more detail on the dataset construction pipeline, such as how each component in Figure 2 was implemented?

  • Could you provide more details about the baseline implementations?

  • GraphSAGE performs worse than MLP on the larger dataset. Do you have an explanation for this result?

  • How were the inputs prepared and fed to the LLM-based methods?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Eoi4

Q1: “Could you include a table comparing the proposed benchmark with existing financial fraud detection benchmarks?”

Appendix A.2 (Table 3) provides a concise comparison of Elliptic, T-Finance, DGraph, and FinFraudBench, focusing on graph schema (homogeneous or heterogeneous), scale, and label/split characteristics. We also clarify that YelpChi and Amazon, although commonly used multi-relational fraud graphs, concern review fraud rather than financial fraud and therefore do not represent routine payment transactions. FinFraudBench specifically targets fraud in routine payment transactions.

Q2: “Could you provide more detail on the dataset construction pipeline, such as how each component in Figure 2 was implemented?”

We agree that the previous Figure 2 could be ambiguous, so we revised it and clarified its stages in the updated manuscript and Appendix A.3. For CreditCard-Fraud, each row in fraudTrain.csv or fraudTest.csv is a transaction, and its recurring customer, card, merchant, category, and location fields define the links to context entities. For BankTrans-Fraud, transaction, user, and card records are linked through their entity IDs, with fraud labels joined to transactions by transaction ID. The builders factorize entity keys into local IDs, create type-specific node tables and feature matrices from non-label fields, and construct seven forward relations: five transaction-to-entity relations and two context relations (customer-card and customer-location). Adding reverse edges yields fourteen directed relation types. Transaction-to-entity edges preserve one edge per transaction, while the context relations are deduplicated; fraud labels are attached only to transaction nodes and excluded from input features. Appendix A.3 also describes the homogeneous and multi-relational projections and the split protocol.

Q3: “Could you provide more details about the baseline implementations?”

Appendix B.1 (Table 6) now reports the principal baseline configurations, including epoch limits, batch sizes, dropout, learning rates, weight decay, architecture, and neighborhood sampling. In Table 1, all non-LLM baselines use a fixed classification threshold of 0.5. The two LLM baselines select their thresholds on the full, imbalanced validation split and hold them fixed across the five balanced mini-tests. Where applicable, checkpoints are selected based on validation AUPRC.

Q4: “GraphSAGE performs worse than MLP on the larger dataset. Do you have an explanation for this result?”

On the balanced BankTrans-Fraud mini-tests, GraphSAGE has lower Macro-F1/Fraud-F1 than the target-feature MLP (0.4286/0.1696 versus 0.6329/0.5133). A possible explanation is that GraphSAGE aggregates sampled neighborhoods on a homogeneous projection, which merges distinct transaction-entity relations. With limited labels, this aggregation may mix in noisy neighbors and dilute transaction-specific fraud signals, reducing its advantage over the MLP.

Q5: “How were the inputs prepared and fed to the LLM-based methods?”

Appendix B.2 now documents the prompt and scoring procedure. Both baselines use frozen Qwen2-7B-Instruct and target-only structured transaction text; available transaction, merchant, and customer fields are included, while labels and direct identifiers are excluded. LLM-ZS receives the target record alone. LLM-ICL-4 adds two benign and two fraudulent demonstrations drawn from labeled training data, with no test-example retrieval. The model scores the candidate “Yes” and “No” sequences by their log probabilities; the normalized scores give the fraud probability. Inference uses the tokenizer's chat template when available and bfloat16.

W1: “The details of dataset construction are insufficiently described, including how the graph structure (nodes and edges) was built.”

As detailed in our response to Q2, Appendix A.3 now describes the node construction, typed relations, edge handling, labels, projections, and splits.

W2: “The paper does not provide implementation details for the evaluated baselines, such as configurations and training settings.”

As detailed in our response to Q3, Appendix B.1 now lists the principal settings used for each baseline.


We have revised the manuscript in response to your suggestions. We greatly appreciate your helpful comments, as they will undoubtedly help us improve the quality of our article. If our response has successfully addressed your concerns and clarified the ambiguities, we respectfully hope that you consider raising the score. Should you have any further questions or require additional clarification, we would be delighted to engage in further discussion. Once again, we sincerely appreciate your time and effort in reviewing our manuscript. Your feedback has been invaluable in improving our research.

AuthorsOpenReview ↗

Has our response resolved your concerns?

Dear Reviewer Eoi4,

We appreciate your insightful and helpful review! Your suggestions on benchmark comparison, graph construction, baseline implementation, and LLM input preparation helped us clarify these parts of the manuscript. We added the corresponding comparison and implementation details to our responses above.

If our response has resolved your concerns and clarified any ambiguities, we respectfully hope you might consider raising the score. Should you have further questions or need additional clarification, we would be happy to discuss them. Thank you again for your time and effort in reviewing our manuscript. Your feedback has been instrumental in improving our research!

Best regards,
Authors

Dear authors,

Thank you for your detailed response. After reading your rebuttal and the revised paper, I still have questions about the dataset, benchmark construction, and the baselines.

  • Could you provide a pseudo-algorithm for each component of the graph construction pipeline?
  • How do you ensure a fair comparison across baselines?

Thank you.

Reviewer_XP2nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FinFraudBench, a transaction-level fraud detection benchmark built by converting two public financial datasets into heterogeneous graphs. It provides six node types, 14 directed relation types, graphs of up to 8.99M nodes and 89.23M edges, and a 2%-labeled training protocol. Comparisons across tabular, homogeneous, multi-relational, fraud-oriented, and heterogeneous models suggest that heterogeneous GNNs are generally competitive, although the main results use balanced test subsets rather than the naturally imbalanced full test sets (Secs. 3–6, Table 2).

Critical strengths
  1. Useful benchmark engineering at scale: two consistent six-type/fourteen-relation graphs (up to 89.23M directed edges), public-source provenance, and explicit limited-label splits can support evaluation beyond single-node-type fraud graphs (Tables 1, 3–4).

  2. Broad empirical coverage: representative methods from five families are compared with both ranking and classification metrics, revealing meaningful differences between AUPRC and fraud recall (Table 2; Fig. 6).

Critical weaknesses
  1. The evaluation does not substantiate the central real-world imbalance claim. The main results are exclusively from 1:1 balanced mini-tests, whereas the full-test fraud rates are 0.3860% and 0.1495%; full-test PR-AUC, precision, recall, and F1 are absent. PR-AUC and precision are prevalence-dependent, so the reported values do not establish deployment effectiveness (Sec. 5.2; Tables 2, 4–6).

  2. The evidence does not isolate the value of heterogeneity. Different families use different graph projections and feature inputs, without controlled type/relation/feature ablations or a strong gradient-boosted tabular baseline. In addition, context-node count aggregates and transductive graph access are insufficiently specified to rule out use of future information (Secs. 3.2, 5–6; Table 3).

Questions for authors
  1. Can you report all core metrics on the naturally imbalanced full test sets, with thresholds selected only on validation data, plus PR-AUC, precision/recall at a fixed alert budget, and false-positive rates? How are thresholds chosen for each method?

  2. Are entity statistics (counts, means, unique-entity counts), normalizers, and graph edges computed exclusively from information available before each prediction? Please specify random vs temporal splits, transductive visibility of validation/test nodes, and entity overlap across splits.

  3. Can you add matched-input/architecture ablations for node types, relation types, and aggregate attributes, together with tuned XGBoost/LightGBM or CatBoost baselines and repeated training seeds (not only five mini-test draws)?

  4. Please clarify source provenance: the CreditCard source is explicitly simulated, while public descriptions of the BankTrans source list separate transaction, user, and card tables. Which original files were used, and how do these facts affect the claimed construction novelty and deployment realism?

6 comments
AuthorsOpenReview ↗

Response to Reviewer XP2n (1/2)

We thank Reviewer XP2n for the careful assessment and constructive suggestions. We respond to each question first, followed by the weaknesses.

Q1: “Can you report core metrics on the naturally imbalanced full test sets, including PR-AUC and precision/recall at a fixed alert budget, and false-positive rates? How are thresholds selected?”

We use balanced mini-tests so that fraud and benign transactions contribute equally to threshold-dependent metrics, and average over five subsets to reduce dependence on a single sample. We additionally report all seven core metrics on the complete test splits at their natural fraud prevalences in Appendix C.2 (Table 8). All baselines reported in Table 8 use a fixed classification threshold of 0.5.

Q2: “Are entity statistics, normalizers, and graph edges computed only from pre-prediction information? Please clarify split types, transductive visibility, and entity overlap.”

The splits are random rather than temporal, and the main evaluation is transductive: non-label graph information and edges from all splits are visible in one shared graph (Section 5.1). Entity attributes and summaries do not use fraud labels and are not computed with a prediction-time cutoff; normalization parameters are also computed across splits rather than from the training split alone. Auxiliary entities are deduplicated and represented by one node in the shared graph, so the same entity can connect transactions from different splits rather than being duplicated as separate split-specific nodes.

Q3: “Can you add matched-input/architecture ablations for node types, relation types, and aggregate attributes, together with tuned XGBoost/LightGBM or CatBoost baselines and repeated training seeds?”

In Section 6.3 (Figure 4), MLP, XGBoost, LightGBM, and CatBoost use source transaction rows enriched with associated-entity attributes and label-independent summaries, without graph edges or message passing. HAN scores 0.7823/0.5660/0.7364 (Macro-F1/Fraud-F1/precision) on CreditCard-Fraud, versus the best tabular scores of 0.7683/0.5385/0.4992; on BankTrans-Fraud, SeHGNN scores 0.8937/0.7877/0.8459, versus 0.8063/0.6132/0.7396. These results indicate that enriched tabular attributes alone do not account for the gains; relational propagation contributes useful information.

Section 6.4 (Figure 5) evaluates HGT and SeHGNN after removing each entity type and its incident relations across train, validation, and test graphs; transaction splits are unchanged, and Macro-F1/Fraud-F1 are measured on the complete test sets. The largest drops vary by dataset and model: category for HGT and customer for SeHGNN on CreditCard-Fraud, and merchant for both models on BankTrans-Fraud.

Given the rebuttal-stage deadline, we evaluated four representative methods with three random seeds on one fixed mini-test subset, reporting mean and standard deviation for Macro-F1, Fraud-F1, AUROC, and AUPRC.

DatasetMethodFamilyMacro-F1Fraud-F1AUROCAUPRC
CreditCard-FraudGCNHomogeneous GNN0.4175±0.00310.4175 \pm 0.00310.1503±0.00550.1503 \pm 0.00550.8251±0.00430.8251 \pm 0.00430.8620±0.00290.8620 \pm 0.0029
CreditCard-FraudR-GCNMulti-relation0.6183±0.01950.6183 \pm 0.01950.4899±0.03160.4899 \pm 0.03160.8825±0.00580.8825 \pm 0.00580.9092±0.00270.9092 \pm 0.0027
CreditCard-FraudPMPFraud-oriented0.6953±0.01490.6953 \pm 0.01490.6118±0.02280.6118 \pm 0.02280.8583±0.00610.8583 \pm 0.00610.8949±0.00500.8949 \pm 0.0050
CreditCard-FraudHGTHeterogeneous0.7908±0.00670.7908 \pm 0.00670.7502±0.00940.7502 \pm 0.00940.8954±0.01800.8954 \pm 0.01800.9206±0.00880.9206 \pm 0.0088
BankTrans-FraudGCNHomogeneous GNN0.6616±0.00990.6616 \pm 0.00990.5623±0.01560.5623 \pm 0.01560.9276±0.00310.9276 \pm 0.00310.9285±0.00140.9285 \pm 0.0014
BankTrans-FraudR-GCNMulti-relation0.8421±0.02290.8421 \pm 0.02290.8175±0.02950.8175 \pm 0.02950.9578±0.00560.9578 \pm 0.00560.9695±0.00390.9695 \pm 0.0039
BankTrans-FraudPMPFraud-oriented0.8560±0.00180.8560 \pm 0.00180.8358±0.00220.8358 \pm 0.00220.9550±0.00440.9550 \pm 0.00440.9663±0.00360.9663 \pm 0.0036
BankTrans-FraudHGTHeterogeneous0.8569±0.00770.8569 \pm 0.00770.8371±0.01000.8371 \pm 0.01000.9753±0.00270.9753 \pm 0.00270.9805±0.00170.9805 \pm 0.0017
AuthorsOpenReview ↗

Response to Reviewer XP2n (2/2)

Q4: “Which original files were used for CreditCard-Fraud and BankTrans-Fraud, and how do the synthetic sources affect the novelty and deployment-realism claims?”

CreditCard-Fraud uses the simulated source files fraudTrain.csv and fraudTest.csv. BankTrans-Fraud uses transactions_data.csv, users_data.csv, and cards_data.csv; transaction rows are linked to user and card records through entity IDs, and fraud labels in train_fraud_labels.json are joined by transaction ID. To make the construction procedure explicit, we added Algorithm 1 in Appendix A.3, which summarizes how the source transaction and entity records are converted into typed nodes, features, relations, and transaction labels. Both public sources are synthetic. We use synthetic sources because transaction-level records from real payment platforms are sensitive and rarely publicly released, while these sources make the benchmark reproducible. The graph schema was informed by risk-control practice at Alipay, Ant Group's mobile-payment platform serving over one billion users, to represent routine-payment entities and relations. The resulting benchmark includes six node types, fourteen directed relation types, up to 8.99M nodes and 89.23M edges, 2% labeled training transactions, and full-test fraud rates of 0.3860% and 0.1495%. These design choices aim to approximate key routine-payment conditions while keeping the released records synthetic. FinFraudBench's contribution extends beyond heterogeneous graph construction: it provides a routine-payment fraud schema informed by risk-control practice and an evaluation protocol reflecting key practical conditions, including large graph scale, limited-label training, and severe class imbalance.

W1: “PR-AUC and precision are prevalence-dependent, so the reported values do not establish deployment effectiveness.”

We address the prevalence concern with the complete-test results described in our response to Q1. These results evaluate methods at each dataset's source-split fraud prevalence.

W2: “The evidence does not isolate the value of heterogeneity, without controlled type/relation/feature ablations or a strong gradient-boosted tabular baseline.”

The added tabular comparisons and node-type ablation are described in our response to Q3. Our response to Q2 clarifies the transductive setting, cross-split visibility, and construction of entity aggregates.


We have revised the manuscript in response to your suggestions. We greatly appreciate your helpful comments, as they will undoubtedly help us improve the quality of our article. If our response has successfully addressed your concerns and clarified the ambiguities, we respectfully hope that you consider raising the score. Should you have any further questions or require additional clarification, we would be delighted to engage in further discussion. Once again, we sincerely appreciate your time and effort in reviewing our manuscript. Your feedback has been invaluable in improving our research.

AuthorsOpenReview ↗

Has our response resolved your concerns?

Dear Reviewer XP2n,

We appreciate your insightful and helpful review! Your comments on evaluation at the original class imbalance and on separating the effects of tabular features from graph structure were particularly helpful. We added full-test results, an entity-enriched tabular comparison, node-type ablations, and a three-seed reference for representative methods in our response above.

If our response has resolved your concerns and clarified any ambiguities, we respectfully hope you might consider raising the score. Should you have further questions or need additional clarification, we would be happy to discuss them. Thank you again for your time and effort in reviewing our manuscript. Your feedback has been instrumental in improving our research!

Best regards,
Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers for their careful and constructive feedback. We are encouraged that reviewers regard FinFraudBench as a useful benchmark resource (55\frac{5}{5} reviewers: XP2n, Eoi4, zRBS, n34Z, and 1wpF), consider its motivation important and practically relevant (35\frac{3}{5} reviewers: Eoi4, zRBS, and 1wpF), describe its baseline evaluation as broad and informative (35\frac{3}{5} reviewers: XP2n, zRBS, and 1wpF), and find its presentation clear and easy to follow (25\frac{2}{5} reviewers: zRBS and 1wpF).

In response to the feedback, we revised the manuscript to clarify the benchmark's scope, data sources, graph construction, split protocols, baseline configurations, and LLM prompting. We added a benchmark comparison, full-test evaluation at source-split prevalence, an entity-enriched tabular comparison, node-type ablations, and an inductive evaluation. We provide point-by-point responses below and thank the reviewers for their helpful suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.