- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper adapts the JEPA recipe to structured event sequences (transactions, purchases, process logs), where each event is a row of heterogeneous fields. The local branch masks fields inside one event and regresses the EMA teacher's latent for the masked fields, which is essentially T-JEPA applied row by row. On top of that the authors add two cross-event mechanisms: a field-aligned attention, where the query for a field of the target event attends only to the same field in the surrounding events and the result is added to the predictor's query behind a zero-initialised gate, and a paired residual auxiliary loss. Pretraining is two-stage (local checkpoint first, then context-augmented). Evaluation is on MBD-mini, AlphaBattle 2.0, X5 Retail and BPI17 with fine-tuning (Table 1), frozen linear probing (Figure 2), a label-efficiency curve and an ablation of the two mechanisms (Table 2).
Critical strengths
- The problem is real and under-served. Latent targets remove the need for a separate head and loss per field type, and whether cross-event context should be field-aligned is a sensible question to ask.
- As far as I know this is the first paper that runs a complete JEPA pipeline (EMA teacher, masked-field latent targets, predictor with mask-token queries) on structured event sequences. The pieces are not new, but the combination has not been reported.
- Zero-initialised gates plus the two-stage schedule are a reasonable way to keep an EMA teacher from collapsing when a new branch is switched on, and Algorithm 1 / Appendix A are detailed enough to follow what is actually computed.
Critical weaknesses
1. I cannot tell where the improvement comes from. The closest baseline is T-JEPA. The "Local only" row of Table 2 is, by the paper's own description, the same objective (per-event masked-field latent prediction with an EMA teacher), yet it already beats T-JEPA by roughly 1 to 3.5 points on every dataset. That gap is larger than everything the two proposed mechanisms add on top of it (under 1.5 points everywhere). So most of the headline gain over the nearest competitor comes from the encoder, the readout or tuning, not from the contribution the paper is about. The field-aligned module is also kept at fine-tuning time, so part of the remaining gain may simply be extra capacity; there is no ablation that drops the module downstream while keeping the pretrained weights.
2. Table 1 and Table 2 do not agree, and there is no variance anywhere. On MBD-mini and AlphaBattle the SE-JEPA numbers in Table 1 are higher than every configuration in Table 2, while on the other two datasets they coincide with different ablation rows. Either Table 1 is a per-dataset pick of the best configuration or the two tables come from different runs; either way it has to be stated. No table reports seeds or standard deviations. EBES reports standard deviations on exactly these datasets that are comparable to the margins claimed here over the best baseline, and on BPI17 the method is below the authors' own CatBoost-on-aggregates. Without seeds I do not know which differences are real.
3. Several baselines are well below published numbers on the same data. The BPI17 baselines (GRU, Transformer, CoLES, MLEM) are 3–6 points below the EBES numbers for the same target, and SE-JEPA only ties the published GRU there. The aggregate+GBDT baselines on X5 are far below the hand-features+LightGBM figure in the CoLES paper. The frozen CoLES probe on MBD-mini is below chance for a mean AUROC, which is almost certainly a bug. Appendix B describes no hyperparameter search for any baseline and one shared downstream configuration for everything, so the proposed method is presumably tuned and the baselines are not.
4. The closest prior work is not cited, and it narrows the novelty considerably. Same-field attention across rows is exactly what Fieldy does (row-wise and column-wise field transformers, masked-field pretraining, motivated by the same sentence about hierarchical models losing field-level patterns across rows), and TabPFN v2, TabICL and NPT all alternate within-row attention with same-column attention across rows. Latent masked prediction on transaction sequences also exists (the CMLM objective of Bazarova et al.). UniTTab handles mixed numeric/categorical targets in observation space with a single loss, which undercuts the introduction's argument for latent targets. Several JEPA variants the introduction should list (TS-JEPA, LeJEPA, Var-JEPA) are missing. What remains new is "add the same-field attention output to the predictor query, excluding the target event". That is a narrow delta, and it is not tested against the obvious alternatives: the same attention inside the encoder (Fieldy) or unrestricted attention over all field tokens across events (STEP).
5. No controlled comparison of the latent target against an observation-space target. Every baseline differs from SE-JEPA in backbone and training recipe at the same time. The experiment that would support the central claim (same encoder, same cross-event modules, same readout, only the target swapped to per-field classification/regression) is absent, and so is any generative / next-event pretraining baseline.
Questions for authors
- Are Tables 1 and 2 from the same runs, and which configuration produced the Table 1 numbers on MBD-mini and AlphaBattle? Please add mean std over at least 5 seeds.
- Why does "Local only" beat T-JEPA by up to 3.5 points when both are per-event masked-field latent prediction? What exactly differs, and can you run T-JEPA with your readout (or vice versa)?
- Please give the full pretraining configuration, and say whether "Local only" is the first-stage checkpoint or a budget-matched run.
- Have you run the same architecture with an observation-space target instead of the latent one, and with the same-field attention placed inside the encoder or replaced by full attention over all field tokens? These are the controls for the paper's two claims.
- Can X5 and BPI17 be reported under the EBES protocol so that they can be compared with the published leaderboard?
8 comments
Response to Reviewer GaEX (Part 2)
W2 & Q1. Result provenance and variability across seeds
Thank you for asking us to clarify the relationship between the main results and the ablations. The caption of Figure 3, Section 4.2, and Appendix C.1 of the original submission report HPO-Val scores as the mean ± one sample standard deviation across three fine-tuning seeds, with the pretrained checkpoint held fixed. We agree that these validation results do not quantify the variability of the test scores in Tables 1 and 2.
1. Relationship between Tables 1 and 2
The original tables drew on different downstream runs. The Table 1 scores of 79.09 on MBD-mini and 76.95 on AlphaBattle both correspond to the full Joint configuration, with same-field interaction and the residual auxiliary objective enabled. On these two datasets, Table 1 and the Joint row of Table 2 used the same model configuration and pretrained weights but different fine-tuning runs, which accounts for the difference in test scores.
To remove this ambiguity, the revision will use the same full Joint configuration on all four datasets: same-field interaction enabled (), a fixed residual weight of , Serial readout, and the same-field module retained downstream. For each dataset, we report all five independent pretraining runs, each followed by fine-tuning. The SE-JEPA row in the main table and the Joint row in the ablation table use the same five test results.
2. Test results across five seeds
We added five-run evaluations. SE-JEPA retains the highest mean test score among the listed methods on MBD-mini, AlphaBattle, and Retail/X5. CatBoost-Agg has the highest mean on BPI17.
Scores are test means ± sample standard deviations (%). SUP5 denotes five independent supervised training runs. FT5 denotes five fine-tuning runs from a fixed pretrained checkpoint. PT5 denotes five independent pretraining runs, each followed by fine-tuning.
| Method / replication protocol | MBD-mini Macro-AUROC | AlphaBattle AUROC | Retail/X5 Accuracy | BPI17 AUROC |
|---|---|---|---|---|
| XGBoost-Agg / SUP5 | 76.56 ± 0.11 | 74.52 ± 0.48 | 46.76 ± 0.66 | 74.79 ± 0.15 |
| CatBoost-Agg / SUP5 | 77.24 ± 0.12 | 74.99 ± 0.13 | 47.72 ± 0.08 | 76.77 ± 0.23 |
| MLP-Agg / SUP5 | 75.19 ± 0.45 | 74.11 ± 0.28 | 46.50 ± 0.10 | 72.24 ± 0.19 |
| GRU / SUP5 | 75.24 ± 0.54 | 73.71 ± 0.23 | 54.74 ± 0.15 | 71.11 ± 0.27 |
| Transformer / SUP5 | 77.41 ± 0.63 | 75.89 ± 0.21 | 53.17 ± 0.19 | 71.62 ± 0.32 |
| CoLES / FT5 | 75.10 ± 0.48 | 74.67 ± 0.09 | 54.42 ± 0.18 | 70.19 ± 0.47 |
| MLEM / FT5 | 77.40 ± 0.21 | 75.99 ± 0.15 | 54.62 ± 0.15 | 70.95 ± 0.18 |
| TabBERT / FT5 | 77.82 ± 0.12 | 76.51 ± 0.22 | 55.12 ± 0.15 | 74.76 ± 0.36 |
| T-JEPA (native mean readout) / FT5 | 74.81 ± 0.16 | 73.23 ± 0.15 | 53.50 ± 0.10 | 71.27 ± 0.71 |
| CF-JEPA / FT5 | 73.09 ± 1.06 | 75.52 ± 0.12 | 51.09 ± 0.24 | 72.07 ± 0.57 |
| HEPA / FT5 | 78.01 ± 0.15 | 75.54 ± 0.09 | 53.26 ± 0.05 | 71.90 ± 0.44 |
| Full SE-JEPA Joint + Serial / PT5 | 78.59 ± 0.14 | 76.97 ± 0.14 | 55.29 ± 0.17 | 75.08 ± 0.10 |
Ablations across five independent pretraining runs (PT5).
| Configuration | MBD-mini Macro-AUROC | AlphaBattle AUROC | Retail/X5 Accuracy | BPI17 AUROC |
|---|---|---|---|---|
| Local | 78.16 ± 0.45 | 75.74 ± 0.42 | 54.55 ± 0.18 | 73.35 ± 0.87 |
| Aux-only | 78.08 ± 0.36 | 75.80 ± 0.38 | 54.55 ± 0.17 | 73.69 ± 1.11 |
| SE-JEPA (Same-field) | 78.66 ± 0.28 | 76.91 ± 0.25 | 55.38 ± 0.10 | 74.00 ± 1.57 |
| SE-JEPA (Joint) | 78.59 ± 0.14 | 76.97 ± 0.14 | 55.29 ± 0.17 | 75.08 ± 0.10 |
Response to Reviewer GaEX (Part 3)
W3 & Q5. Baseline evaluation, tuning fairness, and comparison with EBES
Thank you for raising these issues. We added baseline runs under the EBES protocol, compared methods under matched tuning budgets, and independently checked the CoLES frozen-probe result.
1. Original protocol and EBES. The prediction tasks are the same, but the protocols differ in data splits, input processing, model capacity, readout, and training budget. On BPI17, our case-level holdout yielded 8,798 test examples, compared with 8,833 under EBES. On X5, the corresponding test sets contained 79,734 and 79,541 examples. We therefore reran the baselines on the EBES data and splits using their published EBES configurations and evaluated SE-JEPA with a fixed Joint + Serial configuration. The table reports means ± sample standard deviations over five runs.
| Method / configuration | BPI17 AUROC (%) | X5 Accuracy (%) |
|---|---|---|
| GRU | 75.35 ± 0.16 | 53.75 ± 0.48 |
| Transformer | 75.10 ± 0.58 | 53.29 ± 0.45 |
| CoLES | 74.79 ± 0.86 | 55.35 ± 0.29 |
| MLEM | 75.41 ± 0.19 | 53.81 ± 0.25 |
| SE-JEPA (fixed Joint + Serial) | 75.52 ± 0.33 | 55.12 ± 0.31 |
2. Optimization settings and fairness of hyperparameter search. The original full fine-tuning runs shared a learning rate of , a weight decay of , and validation-based checkpoint selection. Architectures, readouts, and some batch settings were method-specific. For the additional comparison, we specified a limited-budget search for GRU, Transformer, CoLES, MLEM, and SE-JEPA, with six candidates per method per round, a common per-trial budget cap, and selection based solely on validation performance. The revised appendix will report all candidate configurations, the selected configurations, the budgets actually used, and the final results.
3. CoLES frozen probe. With the reconstructed CoLES encoder and evaluation examples held fixed, we reproduced the below-chance result from the original probe. Refitting a linear probe on the same frozen representations raised MBD-mini macro-AUROC to 70.08%. This control indicates that the original probe underestimated the quality of the frozen CoLES representations. We will update Figure 2 using a common probe-selection rule across methods and revise the corresponding conclusions.
W4. Relationship to the closest prior work
Thank you for highlighting these studies. SE-JEPA’s contribution is field-level latent prediction with an event-local teacher and cross-event retrieval in the predictor. The EMA teacher defines each field target from the complete target event. The predictor retrieves the same field from other events of the same entity, excluding the target event from memory. An optional residual objective adds event-level context.
-
How context enters prediction. Fieldy combines row and column context in the encoder to reconstruct masked fields. STEP serializes fields for autoregressive next-token prediction. SE-JEPA uses cross-event context in the predictor to estimate a specified field’s latent representation.
-
What the model predicts. CMLM predicts a masked transaction embedding. UniTTab quantizes numerical targets to use cross-entropy across field types. SE-JEPA predicts a contextualized field embedding without numerical binning.
The revision will discuss these methods alongside TabPFN v2, TabICL, and NPT for modeling sample and feature axes, and related JEPA variants.
Controlled comparisons within SE-JEPA. Entries are mean paired differences ± sample standard deviations across three seeds, in percentage points.
| Comparison | BPI17 AUROC | MBD-mini Macro-AUROC |
|---|---|---|
| Same-field attention: predictor-side − encoder-side | +1.91 ± 0.39 | +0.29 ± 0.42 |
| Predictor-side retrieval: same-field − all-field | +0.06 ± 0.38 | +0.11 ± 0.36 |
| Latent − observed targets (numerical Smooth L1 and categorical cross-entropy) | +1.19 ± 0.16 | −0.08 ± 0.13 |
On BPI17, predictor-side attention and latent targets each improved AUROC in all three seeds. On MBD-mini, attention placement had a smaller effect, and latent targets scored slightly below observed targets. Same-field and all-field retrieval yielded similar scores on both datasets.
Response to Reviewer GaEX (Part 1)
We thank you for your thoughtful review and for recognizing the importance of structured event representation learning, the value of studying the full JEPA pipeline in this setting, and the rationale for our design choices. Your comments identified comparisons and reporting details that help clarify the contribution. Below we address the concerns and questions raised in the review.
Local is an ablation within SE-JEPA’s two-stage training procedure. For each seed, all branches continue training from the same local checkpoint C0 for 5,000 updates. Local disables both cross-event mechanisms and continues within-event latent prediction. It also differs from our original T-JEPA adaptation in masking and downstream readout. The Serial readout incorporates event positions and time gaps and is part of the transfer design described in Appendix A.5.
Gains under a matched readout and training budget. Replacing T-JEPA’s mean readout with Serial raised its mean test score across five fine-tuning runs by 3.53 percentage points on MBD-mini and 1.70 on BPI17. We then matched the starting checkpoint C0 within each seed, the 5,000 additional pretraining updates, and the rule for selecting fine-tuning checkpoints. The table reports results across five independent pretraining seeds.
Under these controls, full SE-JEPA scored 0.44 percentage points higher on MBD-mini and 0.86 higher on BPI17. These gains include the contribution of retaining the pretrained same-field module downstream.
Contribution of the downstream same-field module. Starting from the same Joint checkpoint within each seed, we retained the pretrained encoder and repeated fine-tuning with the same-field module either retained or bypassed throughout fine-tuning and evaluation.
These controls support SE-JEPA’s combined pretraining and transfer design and distinguish the effect of encoder initialization from that of retaining the field-aligned module downstream.