- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the Foundation Event Predictor (FEP), an in-context foundation model for next-event prediction, including both the next event type and the inter-arrival time. It aims to generalize the foundation model to previously unseen, heterogeneous event logs without parameter adaptation. The model decomposes context sequences into event-level history-to-next-event pairs, encodes context histories as keys and target histories as queries via a shared causal Transformer, encodes observed next events as values, and performs learned nonparametric aggregation via cross-attention. FEP is pretrained on 696 real-world event datasets and evaluated on 50 held-out datasets, comparing against per-dataset local models (Transformer, GRU, Neural Hawkes, etc.) and existing foundation models (FM-v2, FIM-PP). Results show FEP outperforms several foundation models and often approaches the best local model.
Critical strengths
- It is an important problem to establish a foundation model generalizing across heterogeneous event logs.
- Pretraining on 696 datasets and testing on 50 held-out real logs is a substantial effort. The experiments are convincing.
Critical weaknesses
- It lacks a clear figure that conveys the model architecture and data flow. The text alone is insufficient to reproduce the model.
- The paper lacks a careful and complete introduction to data curation and the characteristics under different evaluation settings.
- More discussion and explanation are needed to analyze the contributions of the proposed model compared with the baselines and variants. For example, why does FEP-Local lag so far behind Transformer/GRU? Model size ablations show that larger size is worse, but no discussion is given.
Questions for authors
- Could you provide a complete architecture figure to clearly illustrate the data flow in the whole model and the model's architecture?
- Please detail the full preprocessing pipeline for pretraining and testing.
3 comments
Rebuttal
Weakness 3: Insufficient analysis of model contributions, FEP-Local, and model-size ablations
R3: We respectfully disagree that the paper lacks sufficient analysis of model contributions and behavior. Section 4.2 and Appendix B already provide extensive ablation studies, accompanied by detailed analyses and discussions of their findings, addressing several key factors affecting FEP's performance.
(1) Extensive ablation studies and analyses. We systematically investigate context size (Figure 3a), number of pretraining datasets (Table 3a), model dimension and depth (Figure 3b and Table 3b), event-ID representation strategies (Table 6), and time normalization strategies (Table 7). These experiments provide important insights into FEP's behavior, including the benefits of larger context sets and diverse pretraining datasets, the importance of removing persistent event-ID semantics, and the effectiveness of different time normalization strategies. Furthermore, Figure 4 provides dataset-wise comparisons between FEP and FEP-Local, while Appendix B analyzes inference efficiency and scalability (Tables 4–5). These results and their implications are explicitly discussed in the paper.
(2) Why FEP-Local underperforms Transformer/GRU. FEP-Local uses exactly the same architecture, hyperparameters, and training procedure as FEP; the only difference is that it is trained on a single target dataset rather than across heterogeneous datasets. No dataset-specific architecture or hyperparameter optimization is performed for FEP-Local. This controlled comparison is specifically designed to isolate the benefits of cross-dataset pretraining, rather than to establish the best achievable performance of FEP under dataset-specific optimization. In contrast, Transformer and GRU are trained and tuned separately for each dataset. Therefore, the performance gap between FEP-Local and these optimized baselines should not be interpreted as evidence of an inherently weaker architecture. The substantial improvement from FEP-Local to pretrained FEP directly demonstrates the benefits of cross-dataset pretraining, as further supported by the dataset-wise analysis in Figure 4.
(3) Clarification of model-size ablations. The reviewer's statement that larger models perform worse does not accurately reflect our results. Figure 3b shows that increasing model dimension initially yields substantial improvements in both accuracy and NMAE, with performance largely saturating around dimensions 64–128 and deteriorating only when increased further to 256. Similarly, Table 3b shows that increasing model depth from one to a few layers provides modest improvements, while excessive depth (eight layers) degrades performance. These trends are already explicitly discussed in Section 4.2 (under "Model Size"). The results indicate that FEP benefits from sufficient model capacity but does not require an excessively large backbone, with model width having a more pronounced effect than moderate increases in depth.
Thus, the submitted paper already provides systematic ablation studies and substantive analyses addressing the specific issues raised by the reviewer, including the role of cross-dataset pretraining and the non-monotonic effects of model size.
Thank you for the responses. I have read the rebuttal and have no further questions.
Rebuttal
Weakness 1 and Question 1: Missing architecture and data-flow figure
R1: We respectfully clarify that Figure 1 (page 4) of the main paper already provides an explicit visualization of FEP's architecture and complete data flow. This figure illustrates the core methodological contribution of our work: how context and target event sequences are encoded through shared causal self-attention, how context histories and their subsequent events form keys and values, how target histories form queries, and how cross-attention aggregates context information for next-event prediction. The figure is prominently presented in Section 3.2 and directly corresponds to the detailed mathematical formulation that follows. Furthermore, Sections 3.2–3.5 provide the mathematical definitions and model components, while Algorithm 1 in Appendix A explicitly outlines the complete pretraining pipeline, including context and query sampling, causal encoding, cross-attention, loss computation, gradient accumulation, and parameter updates. Appendix A also provides detailed implementation and training configurations. Thus, the requested architecture visualization, data flow, and pretraining pipeline are already explicitly presented and documented in the submitted paper.
Weakness 2 and Question 2: Missing dataset curation, characteristics, and preprocessing details
R2: We respectfully clarify that the submitted paper already provides extensive documentation of dataset curation, preprocessing, dataset characteristics, and experimental procedures, both in the main text and appendices. Specifically:
(1) Dataset curation, preprocessing, and characteristics (Appendix C). Appendix C explicitly describes the dataset sources, curation and preprocessing procedures, including activity-type handling and verification of non-overlap between pretraining and test datasets. Table 9 provides domain-wise statistics for all pretraining and test datasets, including the number of cases, activity types, average trace lengths, and durations. Figure 5 further visualizes the distributions of inter-arrival times, trace durations, activity counts, and trace lengths, comparing the pretraining datasets with easy, moderate, and hard test groups. Furthermore, Table 10 provides a detailed catalog of every test dataset, including their sources, descriptions, statistics, and applicable preprocessing details.
(2) Pretraining and evaluation procedures (Section 4.1 and Appendix A). Section 4.1 explicitly describes the train/validation/test splitting strategy, baseline selection and optimization, evaluation metrics, and dataset difficulty categorization. Appendix A provides further details on model configurations, training settings, baseline implementations, hyperparameter search, and validation procedures, as explicitly referenced in Section 4.1 (Line 294). Moreover, Algorithm 1 in Appendix A presents the complete episodic pretraining procedure, including context and query sampling, causal encoding, cross-attention, gradient accumulation, and parameter updates.
Thus, the requested dataset characteristics, preprocessing methodology, and pretraining/evaluation protocols are already extensively documented in the submitted paper. We believe these descriptions provide a clear basis for understanding and independently implementing the proposed methodology and evaluation procedures.