acceptodds
Under review as a conference paper at ICLR 2027

SCALING TRANSFERABLE BEHAVIORAL REPRESENTATIONS: BRIDGE THE SEMANTIC GAP FROM NEXTEVENT PREDICTION TO DOWNSTREAM TASKS

Abstract

Internet platforms collect abundant behavioral data from a broad population, yet labels for downstream decision tasks are often scarce, delayed, and expensive. A central question is whether large-scale behavioral pre-training can learn representations that transfer across the substantial semantic gap between self-supervised behavior prediction and downstream objectives. We systematically study how pre-training objective, data scale, and model scale shape such cross-task transfer. We introduce BRIDGE (Behavioral Representations for Improving Downstream Generalization and Efficiency), which learns transferable representations of people from large-scale heterogeneous behavioral sequences together with structured features (static or slowly varying person-level attributes, commonly termed userprofile features in recommendation and credit-risk modeling) through autoregressive next-event prediction (NEP). We compare NEP with representative masked, multi-token, distribution-based, and embedding-based objectives. Objective preference depends on scale: embedding-based pre-training leads at the smaller evaluated scale, whereas NEP overtakes it at the larger scale. Despite the weak semantic alignment between NEP and downstream decision objectives, the learned representations transfer effectively to tasks such as credit-risk assessment and payment fraud detection. Across the evaluated scales, downstream out-of-time (OOT) performance improves consistently with both pre-training data volume and model size, while lower pre-training loss is associated with better downstream discrimination. With only 10% of the credit-risk labels, BRIDGE improves the OOT Kolmogorov–Smirnov (KS) statistic by 5.31% relative to end-to-end (E2E) supervised training under the same label budget and even surpasses E2E trained on the full labeled dataset. These results show that scaling behavioral pre-training can substantially improve cross-task transfer and label efficiency, while suggesting that the preferred pre-training objective itself can depend on scale.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Behavioral Pre-training, Cross-task Transfer, User Representation Learning, Scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jZA5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper considers the task of credit-risk assessment and fraud detection on the basis of user behavior sequences. The authors propose the pipeline BRIDGE that has two main steps: (1) pre-training on a large unlabelled data part with a next-event prediction objective and (2) fine-tuning to downstream tasks with scarce ground-truth labels. The paper focuses on transferability between pretraining and downstream stages in view of the semantic gap, the selection of the pretraining objective, and data volume with model sizes. All experiments are done on a private, industrial dataset.

Critical strengths
  1. The authors consider the urgent and important problem of fraud detection and risk assessment leveraging user behavior sequences and structured features, which are ubiquitous in industrial domains.

  2. The experiments are conducted on a large, industrial dataset, which is an important step in working under real-world conditions.

Critical weaknesses
  1. The critical problem with the paper is that it offers little value to other people. All the experiments and numerical findings are obtained for a private dataset. This fact questions the generalization of identified dependencies and trends to other datasets. Beyond this, the obtained conclusions, e.g., that pretraining helps when downstream targets are scarce or that downstream quality increases when the volume of pre-training data is increasing, are general and are known from various other domains like recommendation systems. Moreover, some stated insights, e.g., that the BYB pretraining objective is good for lower pretraining data scales, while NEP is better for larger ones (Table 1), are drawn on the basis of very small metric gains, for which neither standard deviations nor statistical significance are stated. The absence of a formulation of valuable and reusable findings leads me to recommend rejection of this paper.

Less critical weaknesses:

  1. The part of the methodology (Section 3.2.1) is devoted to the description of two architectural changes compared to basic transformer usage. However, there is no discussion on how these changes affect the downstream quality and whether the observed findings will generalize to other architectural choices.

  2. The used Kolmogorov–Smirnov statistic is not introduced either via citation of the paper or by a formula or description.

  3. The methodology part with formulas is hard to follow in some places due to the absence of notation elaboration or inconsistencies (in line 209, HiH_i is not used further in the text; the symbol V\mathcal{V} in line 239 is not introduced; why are the structures of Si−S_i^- in formula 5 and Si+S_i^+ in formula 8 different?).

  4. The conclusion of the paper contains the findings that are obtained in the Appendix (on the benefit of shallower representations for credit-risk prediction).

Questions for authors
  1. How do you select the number of epochs for pretraining for different model sizes? In Appendix C, it is just shown that a greater number of pretraining epochs is better for downstream quality, and the checkpoint from the last epoch is taken.

  2. From formulas 8 and 9, it is not clear how the predictions for different domains are done. Do we predict the tokens for all MM domains at each timestep? Does the formula 9 imply domain-level aggregation?

  3. How do you select the future time horizon T+T^+ for prediction?

Reviewer_XwLQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BRIDGE, which uses large-scale behavioral pre-training with next-event prediction (NEP) to learn representations transferable to credit-risk and fraud detection tasks. It investigates pre-training objectives, data and model scaling, and label efficiency, reporting improved out-of-time performance.

Critical strengths
  1. Practical relevance. The paper addresses the problem of leveraging abundant unlabeled behavioral data for downstream tasks with limited supervision. The reported improvements in credit-risk and fraud detection are encouraging.

  2. Extensive empirical evaluation. The paper compares multiple pre-training objectives, data scales, and model sizes. The label-efficiency experiments and out-of-time evaluations provide useful empirical insights.

Critical weaknesses
  1. Limited evidence for scale-dependent objective selection.

Table 1 compares pre-training objectives at two data scales (1B and 10B). At 10B, NEP outperforms MTP by merely 0.11 KS points and BYB by 0.28 points, without reported uncertainty. It remains unclear whether the observed ranking reversal is statistically reliable.

  1. Insufficient statistical evidence for the label-efficiency claim.

The claim that 5% of labels suffice to outperform full-label E2E relies on a difference of only 0.03 KS points (64.23 vs. 64.20).

3.The presentation could be improved in a few places.

The presentation could be improved in a few places. Section 3.1 could explain the event-level notation and multi-domain ordering operation more explicitly.

Questions for authors
  1. What are the key contributions of BRIDGE beyond existing behavioral pre-training approaches such as BYB and PANTHER?

  2. Could the authors provide an ablation study to clarify the contribution of the gated long-range residual connection introduced in Section 3.2.1?

Reviewer_tX1zOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether large-scale self-supervised pre-training on heterogeneous user behavior sequences transfers to downstream risk prediction despite differences in objectives, label horizons, and data distributions. BRIDGE combines multi-domain event sequences with structured profile features and learns representations through autoregressive next-event prediction, then transfers the embeddings and encoder to full fine-tuning for credit-risk and payment-fraud tasks. Experiments compare five pre-training objectives, vary data scale from approximately 1B to 30B samples and model size from approximately 10M to 100M parameters, and evaluate label efficiency and online deployment. The results suggest useful cross-task transfer, although the main contribution is an empirical scaling study built on largely proprietary data.

Critical strengths
  1. The paper addresses a practically important setting in which behavioral logs are abundant but financial-risk labels are scarce or delayed. The use of out-of-time (OOT) evaluation and an explicit temporal-isolation protocol makes the transfer question meaningful.
  2. The experiments cover complementary dimensions, including five pre-training objectives, data and model scaling, six label fractions, credit-risk prediction, payment-fraud detection, and online results across three business lines.
  3. The global-query encoder gathers information from long histories without applying full self-attention to every event, and the pre-training decoder is removed for downstream use. The label-efficiency result is notable: with 10% of labels, OOT KS is 64.86 versus 61.59 for the matched-budget end-to-end baseline.
Critical weaknesses
  1. Reproducibility is limited by the proprietary data setting. The manuscript reports only approximate data sizes and omits important dataset details such as exact time ranges, sample counts, label definitions, class balance, feature construction, and user/event overlap. Several training details, including exact fine-tuning learning rates, schedules, validation splits, and per-objective tuning budgets, also remain unclear; no code or data access is described.
  2. The scaling and label-efficiency claims lack uncertainty estimates. Although downstream results are averaged over three random seeds, the paper reports no standard deviations, confidence intervals, error bars, or significance tests. Some conclusions depend on very small margins: 5% labels exceed full-label E2E by only 0.03 KS points, while scaling from about 50M to 100M parameters adds only 0.04 KS points. In addition, model width and depth change together, so their individual effects are not isolated.
  3. The evidence for real-world transfer is under-specified. Appendix D reports relative KS improvements and an estimated revenue impact, but omits the online test design, traffic/control assignment, observation period, sample sizes, statistical uncertainty, and revenue-estimation method. The fraud-transfer setup and the temporal relationship between pre-training data and downstream/OOT data are also not detailed enough to independently rule out overlap or leakage.
Questions for authors
  1. Please clarify the chronology and overlap controls among pre-training, downstream training, Test, and OOT data. Can the same users or events appear across stages, and how is future behavior excluded from pre-training relative to each downstream prediction cutoff, including the payment-fraud task?
  2. Please report per-seed results (e.g., mean ± standard deviation or confidence intervals) and significance tests, particularly for the claim that 5% of labels can outperform full-label E2E. Please also clarify the validation split and model-selection procedure used to choose the best fine-tuning epoch.
  3. Please provide the online evaluation protocol for the three business lines, including control or traffic assignment, evaluation duration, sample sizes, uncertainty estimates, and the basis for the revenue estimate. Were all pre-training objectives given comparable tuning and compute budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.