- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper considers the task of credit-risk assessment and fraud detection on the basis of user behavior sequences. The authors propose the pipeline BRIDGE that has two main steps: (1) pre-training on a large unlabelled data part with a next-event prediction objective and (2) fine-tuning to downstream tasks with scarce ground-truth labels. The paper focuses on transferability between pretraining and downstream stages in view of the semantic gap, the selection of the pretraining objective, and data volume with model sizes. All experiments are done on a private, industrial dataset.
Critical strengths
-
The authors consider the urgent and important problem of fraud detection and risk assessment leveraging user behavior sequences and structured features, which are ubiquitous in industrial domains.
-
The experiments are conducted on a large, industrial dataset, which is an important step in working under real-world conditions.
Critical weaknesses
- The critical problem with the paper is that it offers little value to other people. All the experiments and numerical findings are obtained for a private dataset. This fact questions the generalization of identified dependencies and trends to other datasets. Beyond this, the obtained conclusions, e.g., that pretraining helps when downstream targets are scarce or that downstream quality increases when the volume of pre-training data is increasing, are general and are known from various other domains like recommendation systems. Moreover, some stated insights, e.g., that the BYB pretraining objective is good for lower pretraining data scales, while NEP is better for larger ones (Table 1), are drawn on the basis of very small metric gains, for which neither standard deviations nor statistical significance are stated. The absence of a formulation of valuable and reusable findings leads me to recommend rejection of this paper.
Less critical weaknesses:
-
The part of the methodology (Section 3.2.1) is devoted to the description of two architectural changes compared to basic transformer usage. However, there is no discussion on how these changes affect the downstream quality and whether the observed findings will generalize to other architectural choices.
-
The used Kolmogorov–Smirnov statistic is not introduced either via citation of the paper or by a formula or description.
-
The methodology part with formulas is hard to follow in some places due to the absence of notation elaboration or inconsistencies (in line 209, is not used further in the text; the symbol in line 239 is not introduced; why are the structures of in formula 5 and in formula 8 different?).
-
The conclusion of the paper contains the findings that are obtained in the Appendix (on the benefit of shallower representations for credit-risk prediction).
Questions for authors
-
How do you select the number of epochs for pretraining for different model sizes? In Appendix C, it is just shown that a greater number of pretraining epochs is better for downstream quality, and the checkpoint from the last epoch is taken.
-
From formulas 8 and 9, it is not clear how the predictions for different domains are done. Do we predict the tokens for all domains at each timestep? Does the formula 9 imply domain-level aggregation?
-
How do you select the future time horizon for prediction?