acceptodds
Under review as a conference paper at ICLR 2027

MEDT: Delayed-Feedback Expert Selection for Streaming Fraud Detection

Abstract

Streaming fraud detection must adapt to evolving transaction behavior while recent labels may remain unavailable under delayed supervision. Earlier fraud characteristics may persist, coexist, or reappear, creating the problem of deciding which retained historical models should participate before the newest supervision arrives. We propose Mixed Expert Decision Trees (MEDT), which retains temporally differentiated experts, reassesses predictive competence from observable delayed feedback, and selects high-scoring eligible experts for committee prediction. A frozen historical anchor provides a standing reference, and committee probabilities are combined through anchor-stabilized probability fusion. On the IEEE-CIS-based W26 controlled-perturbation stream, MEDT achieves a PR-AUC of 0.4628, 17.0% above the strongest evaluated online baseline. Removing the temporal-expert component reduces PR-AUC by 0.0436, while replacing soft fusion with hard voting reduces it by 0.1023; both contrasts have Holm-corrected p<0.05 in the planned within-stream paired tests. These results support the temporal-expert component as implemented and soft probability fusion, without establishing a uniquely optimal competence or selection rule.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
streaming fraud detection, delayed supervision, temporal experts, competence reassessment, expert selection, historical model retention, adaptive ensembles

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7tqjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MEDT proposes an engineering-reasonable expert committee framework for streaming fraud detection under delayed supervision, and achieves the best PR-AUC among online methods on W26. However, its baselines are outdated, its introduction and literature review are insufficient, it is validated on only a single dataset, its core components lack novelty and independent significant gains, and its inference latency of 7.96 seconds severely contradicts its streaming-oriented positioning.

Critical strengths

MEDT retains temporally differentiated experts, reassesses predictive competence from observable delayed feedback, and selects high-scoring eligible experts for committee prediction. A frozen historical anchor provides a standing reference, and committee probabilities are combined through anchor-stabilized probability fusion. On the IEEE-CIS-based W26 controlled-perturbation stream, MEDT achieves a PR-AUC of 0.4628, 17.0% above the strongest evaluated online baseline. Removing the temporal-expert component reduces PR-AUC by 0.0436, while replacing soft fusion with hard voting reduces it by 0.1023; both contrasts have Holm-corrected p < 0.05 in the planned within-stream paired tests.

Critical weaknesses
  1. The baseline selection is severely outdated, which directly weakens the persuasiveness of the claim "17.0% above the strongest evaluated online baseline."
  2. The introduction is poorly presented, and the literature review is insufficient, making it difficult to read.
  3. The dataset validation is limited to a single dataset, lacking real-world generalizability.
  4. The motivation of the proposed method is not sufficiently clear, and the core components of MEDT have similar ideas in the literature after 2020.
  5. Table 15 shows that MEDT has a Predict latency of 7.96s, the highest among all methods in the table, two orders of magnitude higher than XGBoost (0.06s) and about 26× higher than Streaming HGB (0.30s), which further weakens its contribution given the high sensitivity to latency.
Questions for authors
  1. Add online / delayed-feedback / continual fraud detection baselines from 2022–2025 to re-verify whether the 17.0% lead still holds.
  2. Demonstrate the independent significant gain of at least one core component: at least one of the competence estimator, selection rule, or anchor must significantly outperform its alternative.
  3. Validate on multiple public fraud datasets, or at least provide cross-dataset / cross-time-period generalization experiments to show that the conclusions do not depend on the specific construction of W26.
Reviewer_8qdROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper proposes an online approach for fraud detection under delayed feedback. Given a set of experts, which are gradient boosted models trained on temporal subsets of data, the approach selects the top N scoring ones, between those for which feedback already allows evaluation, plus a base expert trained on the initial data (which the authors call "anchor").

Critical strengths
  • The proposed evaluation setup built on top of the IEEE-CIS dataset strikes a good balance between realistic enough to yield useful practical insights but simple enough to be used as benchmark in an academic paper.
  • The choice of metrics seems sensible.
  • The ablations are interesting and informative of the hyperparameter sensitivities.
Critical weaknesses
  • One of my main issues with the paper is motivation, particularly around the use of online learning. While I understand the general need for adaptation, given the adversarial nature of fraud detection, the experiments show that all online learning approaches underperform a simple weekly model retraining of an XGBoost model. Training these models is relatively cheap, and seems simpler in practice than the proposed approach or other online setups.
  • The experiments use a single dataset and the nature of the task and temporal-based setup precludes cross-validation. While I fully understand the challenges around good quality publicly available fraud detection data, using a single evaluation datapoint raises serious concerns of how trustworthy and generalizable can the conclusions be. This is particularly concerning if there was any iteration in arriving at the method and hyper-parameters, where the full pipeline was run and decisions informed by peeking at the single test evaluation.
  • The paper does not really introduce any novel technique or method. It combines existing well known ideas, in a straightforward way, to provide a sensible approach to a practical problem. As such, and given the particularly restricted domain of application, I am not sure if it is a good fit for this conference, rather than a more targeted venue.
  • While the writing is clear enough to understand the method, I found it repetitive and dense, particularly in the introduction, related work and experiments. The experiments chapter in particular reads like an unfiltered dump of information reiterating numbers that are already reported in tables. There is also not a single diagram of the method in the entire paper which could really help a reader better understand the approach at a glance. A diagram of the evaluation setup would also be helpful. I note that the authors had plenty of space left-over for these. As such the paper feels somewhat unpolished/unfinished.
Questions for authors
  • Why should practitioners opt to use MEDT over simple periodic model retrainings?
Reviewer_NnknOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MEDT for streaming fraud detection under delayed supervision. MEDT retains frozen experts trained on different temporal segments, reassesses their competence using delayed label-visible feedback, and selects a subset for prediction. Experiments on a controlled 26-week stream constructed from IEEE-CIS report higher PR-AUC than the evaluated online baselines. Ablations support the temporal-expert component and soft fusion, but do not establish independent benefits for the specific competence estimator, selection rule, or anchor.

Critical strengths
  1. Relevant delayed-supervision problem and coherent design. Delayed fraud labels and potentially reusable historical knowledge are practically relevant considerations. MEDT provides a clear pipeline for retaining temporal experts, reassessing them using observable feedback, and forming a prediction committee. The chronological protocol explicitly constrains label-dependent operations.

  2. Useful evaluation and appropriately qualified conclusions. MEDT improves over several online baselines, and the experiments include component ablations, parameter sensitivity, and computational measurements. The paper explicitly acknowledges that its ablations do not establish unique benefits for the default competence estimator, selection rule, or anchor. This distinction between observed results and unproven mechanisms improves the credibility of the empirical discussion.

Critical weaknesses
  1. Insufficient justification for the online-learning formulation. The paper does not sufficiently explain why the considered setting requires online learning rather than periodic retraining on all label-visible historical data. The XGBoost retraining baseline respects the same label-availability constraints and achieves higher PR-AUC (0.4829 versus 0.4628) with lower measured weekly runtime (11.77 versus 68.27 seconds). Merely categorizing XGBoost as a batch reference does not establish why it is unsuitable for the target scenario. This leaves the practical motivation for MEDT insufficiently justified and is my main concern.

  2. Insufficient evidence for the methodological contribution. Uniform recent-N selection achieves PR-AUC 0.4633, compared with 0.4628 for full MEDT, while raw-precision scoring reaches 0.4782; neither comparison establishes an advantage for the proposed default. Moreover, the frozen anchor alone achieves 0.4192, already above the strongest evaluated online baseline at 0.3955, making matched-base-learner controls particularly important.

Questions for authors

Could the authors clarify why the considered setting calls for an online-learning formulation rather than periodic retraining on all label-visible historical data? The XGBoost retraining baseline respects the same label-availability constraints and achieves higher PR-AUC with lower measured runtime. What concrete deployment constraints make this alternative unsuitable, and how are these constraints reflected in the experimental setup?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.