acceptodds
Under review as a conference paper at ICLR 2027

SE-JEPA: Learning Representations of Structured Event Sequences via Latent Prediction

Abstract

Structured event data contains within-event field dependencies and cross-event contextual relationships that can provide complementary information for representation learning. A key challenge is to integrate information from related events into field-level predictive learning. We propose the Structured-Event Joint-Embedding Predictive Architecture (SE-JEPA), a self-supervised framework that combines local field modeling with cross-event context through masked-field latent prediction. SE-JEPA constructs within-event representations using field-specific encodings and retrieves field-aligned context through same-field cross-event attention. An event-level residual auxiliary objective further uses pooled event representations to learn context-induced corrections to local predictions. The two mechanisms optimize a shared encoder and predictor using latent targets supplied by an exponential moving-average teacher. Experiments on four public structured-event benchmarks covering diverse downstream tasks demonstrate the effectiveness of SE-JEPA under frozen linear evaluation and full fine-tuning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-Supervised Learning, Structured Event Sequences, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GaEXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adapts the JEPA recipe to structured event sequences (transactions, purchases, process logs), where each event is a row of heterogeneous fields. The local branch masks fields inside one event and regresses the EMA teacher's latent for the masked fields, which is essentially T-JEPA applied row by row. On top of that the authors add two cross-event mechanisms: a field-aligned attention, where the query for a field of the target event attends only to the same field in the surrounding events and the result is added to the predictor's query behind a zero-initialised gate, and a paired residual auxiliary loss. Pretraining is two-stage (local checkpoint first, then context-augmented). Evaluation is on MBD-mini, AlphaBattle 2.0, X5 Retail and BPI17 with fine-tuning (Table 1), frozen linear probing (Figure 2), a label-efficiency curve and an ablation of the two mechanisms (Table 2).

Critical strengths
  • The problem is real and under-served. Latent targets remove the need for a separate head and loss per field type, and whether cross-event context should be field-aligned is a sensible question to ask.
  • As far as I know this is the first paper that runs a complete JEPA pipeline (EMA teacher, masked-field latent targets, predictor with mask-token queries) on structured event sequences. The pieces are not new, but the combination has not been reported.
  • Zero-initialised gates plus the two-stage schedule are a reasonable way to keep an EMA teacher from collapsing when a new branch is switched on, and Algorithm 1 / Appendix A are detailed enough to follow what is actually computed.
Critical weaknesses

1. I cannot tell where the improvement comes from. The closest baseline is T-JEPA. The "Local only" row of Table 2 is, by the paper's own description, the same objective (per-event masked-field latent prediction with an EMA teacher), yet it already beats T-JEPA by roughly 1 to 3.5 points on every dataset. That gap is larger than everything the two proposed mechanisms add on top of it (under 1.5 points everywhere). So most of the headline gain over the nearest competitor comes from the encoder, the readout or tuning, not from the contribution the paper is about. The field-aligned module is also kept at fine-tuning time, so part of the remaining gain may simply be extra capacity; there is no ablation that drops the module downstream while keeping the pretrained weights.

2. Table 1 and Table 2 do not agree, and there is no variance anywhere. On MBD-mini and AlphaBattle the SE-JEPA numbers in Table 1 are higher than every configuration in Table 2, while on the other two datasets they coincide with different ablation rows. Either Table 1 is a per-dataset pick of the best configuration or the two tables come from different runs; either way it has to be stated. No table reports seeds or standard deviations. EBES reports standard deviations on exactly these datasets that are comparable to the margins claimed here over the best baseline, and on BPI17 the method is below the authors' own CatBoost-on-aggregates. Without seeds I do not know which differences are real.

3. Several baselines are well below published numbers on the same data. The BPI17 baselines (GRU, Transformer, CoLES, MLEM) are 3–6 points below the EBES numbers for the same target, and SE-JEPA only ties the published GRU there. The aggregate+GBDT baselines on X5 are far below the hand-features+LightGBM figure in the CoLES paper. The frozen CoLES probe on MBD-mini is below chance for a mean AUROC, which is almost certainly a bug. Appendix B describes no hyperparameter search for any baseline and one shared downstream configuration for everything, so the proposed method is presumably tuned and the baselines are not.

4. The closest prior work is not cited, and it narrows the novelty considerably. Same-field attention across rows is exactly what Fieldy does (row-wise and column-wise field transformers, masked-field pretraining, motivated by the same sentence about hierarchical models losing field-level patterns across rows), and TabPFN v2, TabICL and NPT all alternate within-row attention with same-column attention across rows. Latent masked prediction on transaction sequences also exists (the CMLM objective of Bazarova et al.). UniTTab handles mixed numeric/categorical targets in observation space with a single loss, which undercuts the introduction's argument for latent targets. Several JEPA variants the introduction should list (TS-JEPA, LeJEPA, Var-JEPA) are missing. What remains new is "add the same-field attention output to the predictor query, excluding the target event". That is a narrow delta, and it is not tested against the obvious alternatives: the same attention inside the encoder (Fieldy) or unrestricted attention over all field tokens across events (STEP).

5. No controlled comparison of the latent target against an observation-space target. Every baseline differs from SE-JEPA in backbone and training recipe at the same time. The experiment that would support the central claim (same encoder, same cross-event modules, same readout, only the target swapped to per-field classification/regression) is absent, and so is any generative / next-event pretraining baseline.

Questions for authors
  1. Are Tables 1 and 2 from the same runs, and which configuration produced the Table 1 numbers on MBD-mini and AlphaBattle? Please add mean std over at least 5 seeds.
  2. Why does "Local only" beat T-JEPA by up to 3.5 points when both are per-event masked-field latent prediction? What exactly differs, and can you run T-JEPA with your readout (or vice versa)?
  3. Please give the full pretraining configuration, and say whether "Local only" is the first-stage checkpoint or a budget-matched run.
  4. Have you run the same architecture with an observation-space target instead of the latent one, and with the same-field attention placed inside the encoder or replaced by full attention over all field tokens? These are the controls for the paper's two claims.
  5. Can X5 and BPI17 be reported under the EBES protocol so that they can be compared with the published leaderboard?
8 comments
AuthorsOpenReview ↗

Response to Reviewer GaEX (Part 1)

We thank you for your thoughtful review and for recognizing the importance of structured event representation learning, the value of studying the full JEPA pipeline in this setting, and the rationale for our design choices. Your comments identified comparisons and reporting details that help clarify the contribution. Below we address the concerns and questions raised in the review.

W1. Sources of the performance gains

Local is an ablation within SE-JEPA’s two-stage training procedure. For each seed, all branches continue training from the same local checkpoint C0C_0 for 5,000 updates. Local disables both cross-event mechanisms and continues within-event latent prediction. It also differs from our original T-JEPA adaptation in masking and downstream readout. The Serial readout incorporates event positions and time gaps and is part of the transfer design described in Appendix A.5.

Gains under a matched readout and training budget. Replacing T-JEPA’s mean readout with Serial raised its mean test score across five fine-tuning runs by 3.53 percentage points on MBD-mini and 1.70 on BPI17. We then matched the starting checkpoint C0C_0 within each seed, the 5,000 additional pretraining updates, and the rule for selecting fine-tuning checkpoints. The table reports results across five independent pretraining seeds.

Configuration (Serial readout for both)MBD-mini Macro-AUROC (%)BPI17 AUROC (%)
Continued pretraining with the T-JEPA objective78.14 ± 0.4274.22 ± 0.30
SE-JEPA Joint, full transfer78.59 ± 0.1475.08 ± 0.10

Under these controls, full SE-JEPA scored 0.44 percentage points higher on MBD-mini and 0.86 higher on BPI17. These gains include the contribution of retaining the pretrained same-field module downstream.

Contribution of the downstream same-field module. Starting from the same Joint checkpoint within each seed, we retained the pretrained encoder and repeated fine-tuning with the same-field module either retained or bypassed throughout fine-tuning and evaluation.

Downstream configuration after Joint pretrainingMBD-mini Macro-AUROC (%)BPI17 AUROC (%)
Encoder retained, same-field module bypassed78.07 ± 0.1773.66 ± 1.05
Encoder and pretrained same-field module retained78.59 ± 0.1475.08 ± 0.10
Paired difference, retained − bypassed (percentage points)+0.52 ± 0.23+1.42 ± 1.05

These controls support SE-JEPA’s combined pretraining and transfer design and distinguish the effect of encoder initialization from that of retaining the field-aligned module downstream.

AuthorsOpenReview ↗

Response to Reviewer GaEX (Part 2)

W2 & Q1. Result provenance and variability across seeds

Thank you for asking us to clarify the relationship between the main results and the ablations. The caption of Figure 3, Section 4.2, and Appendix C.1 of the original submission report HPO-Val scores as the mean ± one sample standard deviation across three fine-tuning seeds, with the pretrained checkpoint held fixed. We agree that these validation results do not quantify the variability of the test scores in Tables 1 and 2.

1. Relationship between Tables 1 and 2

The original tables drew on different downstream runs. The Table 1 scores of 79.09 on MBD-mini and 76.95 on AlphaBattle both correspond to the full Joint configuration, with same-field interaction and the residual auxiliary objective enabled. On these two datasets, Table 1 and the Joint row of Table 2 used the same model configuration and pretrained weights but different fine-tuning runs, which accounts for the difference in test scores.

To remove this ambiguity, the revision will use the same full Joint configuration on all four datasets: same-field interaction enabled (a=1a=1), a fixed residual weight of λ=0.05\lambda=0.05, Serial readout, and the same-field module retained downstream. For each dataset, we report all five independent pretraining runs, each followed by fine-tuning. The SE-JEPA row in the main table and the Joint row in the ablation table use the same five test results.

2. Test results across five seeds

We added five-run evaluations. SE-JEPA retains the highest mean test score among the listed methods on MBD-mini, AlphaBattle, and Retail/X5. CatBoost-Agg has the highest mean on BPI17.

Scores are test means ± sample standard deviations (%). SUP5 denotes five independent supervised training runs. FT5 denotes five fine-tuning runs from a fixed pretrained checkpoint. PT5 denotes five independent pretraining runs, each followed by fine-tuning.

Method / replication protocolMBD-mini Macro-AUROCAlphaBattle AUROCRetail/X5 AccuracyBPI17 AUROC
XGBoost-Agg / SUP576.56 ± 0.1174.52 ± 0.4846.76 ± 0.6674.79 ± 0.15
CatBoost-Agg / SUP577.24 ± 0.1274.99 ± 0.1347.72 ± 0.0876.77 ± 0.23
MLP-Agg / SUP575.19 ± 0.4574.11 ± 0.2846.50 ± 0.1072.24 ± 0.19
GRU / SUP575.24 ± 0.5473.71 ± 0.2354.74 ± 0.1571.11 ± 0.27
Transformer / SUP577.41 ± 0.6375.89 ± 0.2153.17 ± 0.1971.62 ± 0.32
CoLES / FT575.10 ± 0.4874.67 ± 0.0954.42 ± 0.1870.19 ± 0.47
MLEM / FT577.40 ± 0.2175.99 ± 0.1554.62 ± 0.1570.95 ± 0.18
TabBERT / FT577.82 ± 0.1276.51 ± 0.2255.12 ± 0.1574.76 ± 0.36
T-JEPA (native mean readout) / FT574.81 ± 0.1673.23 ± 0.1553.50 ± 0.1071.27 ± 0.71
CF-JEPA / FT573.09 ± 1.0675.52 ± 0.1251.09 ± 0.2472.07 ± 0.57
HEPA / FT578.01 ± 0.1575.54 ± 0.0953.26 ± 0.0571.90 ± 0.44
Full SE-JEPA Joint + Serial / PT578.59 ± 0.1476.97 ± 0.1455.29 ± 0.1775.08 ± 0.10

Ablations across five independent pretraining runs (PT5).

ConfigurationMBD-mini Macro-AUROCAlphaBattle AUROCRetail/X5 AccuracyBPI17 AUROC
Local78.16 ± 0.4575.74 ± 0.4254.55 ± 0.1873.35 ± 0.87
Aux-only78.08 ± 0.3675.80 ± 0.3854.55 ± 0.1773.69 ± 1.11
SE-JEPA (Same-field)78.66 ± 0.2876.91 ± 0.2555.38 ± 0.1074.00 ± 1.57
SE-JEPA (Joint)78.59 ± 0.1476.97 ± 0.1455.29 ± 0.1775.08 ± 0.10
AuthorsOpenReview ↗

Response to Reviewer GaEX (Part 3)

W3 & Q5. Baseline evaluation, tuning fairness, and comparison with EBES

Thank you for raising these issues. We added baseline runs under the EBES protocol, compared methods under matched tuning budgets, and independently checked the CoLES frozen-probe result.

1. Original protocol and EBES. The prediction tasks are the same, but the protocols differ in data splits, input processing, model capacity, readout, and training budget. On BPI17, our case-level holdout yielded 8,798 test examples, compared with 8,833 under EBES. On X5, the corresponding test sets contained 79,734 and 79,541 examples. We therefore reran the baselines on the EBES data and splits using their published EBES configurations and evaluated SE-JEPA with a fixed Joint + Serial configuration. The table reports means ± sample standard deviations over five runs.

Method / configurationBPI17 AUROC (%)X5 Accuracy (%)
GRU75.35 ± 0.1653.75 ± 0.48
Transformer75.10 ± 0.5853.29 ± 0.45
CoLES74.79 ± 0.8655.35 ± 0.29
MLEM75.41 ± 0.1953.81 ± 0.25
SE-JEPA (fixed Joint + Serial)75.52 ± 0.3355.12 ± 0.31

2. Optimization settings and fairness of hyperparameter search. The original full fine-tuning runs shared a learning rate of 3×10−43 \times 10^{-4}, a weight decay of 10−410^{-4}, and validation-based checkpoint selection. Architectures, readouts, and some batch settings were method-specific. For the additional comparison, we specified a limited-budget search for GRU, Transformer, CoLES, MLEM, and SE-JEPA, with six candidates per method per round, a common per-trial budget cap, and selection based solely on validation performance. The revised appendix will report all candidate configurations, the selected configurations, the budgets actually used, and the final results.

3. CoLES frozen probe. With the reconstructed CoLES encoder and evaluation examples held fixed, we reproduced the below-chance result from the original probe. Refitting a linear probe on the same frozen representations raised MBD-mini macro-AUROC to 70.08%. This control indicates that the original probe underestimated the quality of the frozen CoLES representations. We will update Figure 2 using a common probe-selection rule across methods and revise the corresponding conclusions.

W4. Relationship to the closest prior work

Thank you for highlighting these studies. SE-JEPA’s contribution is field-level latent prediction with an event-local teacher and cross-event retrieval in the predictor. The EMA teacher defines each field target from the complete target event. The predictor retrieves the same field from other events of the same entity, excluding the target event from memory. An optional residual objective adds event-level context.

  • How context enters prediction. Fieldy combines row and column context in the encoder to reconstruct masked fields. STEP serializes fields for autoregressive next-token prediction. SE-JEPA uses cross-event context in the predictor to estimate a specified field’s latent representation.

  • What the model predicts. CMLM predicts a masked transaction embedding. UniTTab quantizes numerical targets to use cross-entropy across field types. SE-JEPA predicts a contextualized field embedding without numerical binning.

The revision will discuss these methods alongside TabPFN v2, TabICL, and NPT for modeling sample and feature axes, and related JEPA variants.

Controlled comparisons within SE-JEPA. Entries are mean paired differences ± sample standard deviations across three seeds, in percentage points.

ComparisonBPI17 AUROCMBD-mini Macro-AUROC
Same-field attention: predictor-side − encoder-side+1.91 ± 0.39+0.29 ± 0.42
Predictor-side retrieval: same-field − all-field+0.06 ± 0.38+0.11 ± 0.36
Latent − observed targets (numerical Smooth L1 and categorical cross-entropy)+1.19 ± 0.16−0.08 ± 0.13

On BPI17, predictor-side attention and latent targets each improved AUROC in all three seeds. On MBD-mini, attention placement had a smaller effect, and latent targets scored slightly below observed targets. Same-field and all-field retrieval yielded similar scores on both datasets.

Reviewer_3W8XOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SE-JEPA, a self-supervised representation-learning framework for structured event sequences. It extends masked-field latent prediction from single records/fields to sequences by keeping field-level JEPA targets from an EMA teacher while enriching the predictor with cross-event context at two granularities: (i) same-field cross-event attention, which retrieves the same field position from other valid events in the sequence and is retained for downstream encoding; and (ii) event-level contextual auxiliary learning, which mean-pools context events and trains a paired residual objective so that the context-induced prediction change matches the detached local prediction error. The total loss is the main latent-prediction loss plus a weighted auxiliary residual loss. Experiments on four benchmarks—MBD-mini, AlphaBattle, Retail/X5, and BPI17—report full fine-tuning, frozen linear evaluation, label efficiency, component ablations, and a short gradient analysis of the residual objective. SE-JEPA ranks first on three of four datasets under both frozen and fine-tuned evaluation, with especially large frozen gains on AlphaBattle; BPI17 is the main exception, where CatBoost-Agg and MLEM are stronger in the fine-tuned and frozen settings, respectively.

Critical strengths

Strength 1: Clear motivation and reasonable methodological design. The paper combines within-event field dependencies with cross-event contextual information. By introducing field-aligned cross-event attention while preserving the JEPA latent prediction objective, the method provides a simple and intuitive way to exploit the semantic structure of event fields. The ablation results show that same-field attention consistently improves performance across all four benchmarks.

Strength 2: Encouraging experimental results across multiple evaluation settings. The paper evaluates the method on four benchmarks covering different application scenarios, including frozen linear evaluation, full fine-tuning, label-efficiency analysis, and component ablations. SE-JEPA achieves competitive performance across these settings, with a particularly notable 5.01-point AUROC improvement over the strongest reported baseline on AlphaBattle under frozen evaluation. The component ablations also provide useful insights into the different contributions of the two context mechanisms.

Critical weaknesses

Weaknesses 1: The contribution is likely to be viewed as incremental and the empirical claim is conditional on a limited baseline/task set. The method is essentially a careful combination of masked-field JEPA, cross-event same-field attention, and a residual auxiliary objective; all downstream tasks are sequence-level classification on four benchmarks. Several very relevant recent event-sequence/foundation-style models are cited but not compared, including PRAGMA, PANTHER, and FlexTPP, while MLEM coverage appears incomplete across datasets. This does not invalidate the reported results, but it limits the evidence supporting SE-JEPA's competitiveness against recent structured-event representation learning methods. Weaknesses 2: The distinctive benefits of the two core mechanisms are not sufficiently established. The ablation results show that same-field attention consistently improves performance, whereas the benefits of event-level residual auxiliary learning are dataset-dependent. The paper does not directly demonstrate whether the residual objective offers advantages over standard contextual prediction losses, nor does it compare field-aligned attention with more general cross-event attention. Furthermore, pretraining excludes the target event from the attention memory, while downstream encoding includes the query event itself. This discrepancy may affect the interpretation of the improvements attributed to cross-event contextual information.

Questions for authors

Question 1: Necessity of the Residual Auxiliary Objective The paper introduces a paired residual objective to learn context-induced corrections to local predictions. However, the ablation results show that the auxiliary branch alone provides limited or negative improvements, while its benefits in the joint configuration are mainly observed on AlphaBattle and BPI17. Could the authors compare the proposed residual objective with directly supervising the event-contextual prediction using the EMA teacher targets, while keeping the event-level attention module, architecture, and training settings consistent? Since the two objectives have the same forward loss value but different gradients, this comparison would help clarify whether the proposed residual formulation provides an advantage beyond standard contextual prediction.

Question 2: Necessity of Field-Aligned Cross-Event Attention The proposed same-field attention mechanism restricts each target field to retrieving representations from the corresponding field positions in other events. While the ablation results demonstrate consistent improvements over local-only prediction, they do not establish whether these gains specifically arise from field alignment or from incorporating cross-event attention more generally. Could the authors compare same-field attention with a general cross-event attention mechanism that allows each target field to attend to all field representations from other events, under comparable model capacity and training budgets? This would help clarify the contribution of explicitly preserving field correspondence.

Question 3: Self-Inclusion in Downstream Same-Field Attention During pretraining, the target event is explicitly excluded from the cross-event context memory. However, during downstream encoding, the same-field attention memory includes all valid events, including the query event itself. This allows the attention module to retrieve the query field's own representation, potentially reducing its reliance on information from other events. Could the authors explain the motivation for this difference and provide a downstream ablation that excludes the query event from the attention memory? Reporting the learned same-field gate γsf\gamma_{sf} and attention-weight statistics, particularly the attention assigned to the query event itself, would also help clarify how the module uses cross-event information.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3W8X (Part 1)

Thank you for your thoughtful review and important questions. We appreciate your recognition of the clear motivation, sound design, and encouraging empirical results of our work. Below we address the concerns and questions raised in the review.

W1. Methodological contribution, baseline coverage, and task scope

Core contribution. SE-JEPA integrates within-event field dependencies and cross-event information into field-level latent prediction for structured event sequences. The EMA teacher encodes only the unmasked target event to define the representation to predict for each field. Given the target field’s identity, the predictor retrieves representations of the same field from other available events of the same entity, excluding the entire target event from retrieval memory. Cross-event context thus informs the prediction of a specific field, while the target event itself defines the supervision.

Additional baselines. We evaluated adapted versions of PANTHER and FlexTPP and completed the full-recipe MLEM comparison across four sequence classification benchmarks. Under the original protocol, SE-JEPA achieved a higher mean test score than each of these baselines on all four benchmarks.

MethodMBD-mini Macro-AUROCAlphaBattle AUROCRetail/X5 AccuracyBPI17 AUROC
PANTHER (adapted, PT5)71.53 ± 0.5672.33 ± 0.6130.53 ± 0.1658.40 ± 1.13
FlexTPP (adapted, PT5)77.24 ± 0.2274.80 ± 0.3551.02 ± 0.1972.63 ± 1.05
MLEM (full training recipe, FT5)77.40 ± 0.2175.99 ± 0.1554.62 ± 0.1570.95 ± 0.18
SE-JEPA (Joint + Serial, PT5)78.59 ± 0.1476.97 ± 0.1455.29 ± 0.1775.08 ± 0.10

Scores are test means ± sample standard deviations (%). PT5 denotes five independent pretraining runs, each followed by full fine-tuning. FT5 denotes five full fine-tuning runs from a fixed pretrained checkpoint.

W2 & Q1. Is the residual auxiliary objective necessary?

Thank you for this constructive question. Our additional experiments distinguish the more consistent contribution of field-aligned context modeling from the dataset-dependent contribution of residual auxiliary learning.

The two mechanisms serve different roles. Field-aligned attention provides cross-event context and remains active during fine-tuning. The residual objective uses event-level context to correct local predictions during pretraining. Its auxiliary branch is removed before fine-tuning and therefore adds no downstream model capacity. The residual objective is optional, and its use can be guided by validation performance. As described in Section 3.3, Eq. (8), setting λ=0\lambda=0 removes the auxiliary pretraining cost while retaining same-field attention.

Controlled comparison. We compared the paired residual objective with direct prediction of the EMA teacher target (Direct-MSE). Both used the same event-level attention, architecture, initialization, training budget, and auxiliary weight (λ=0.05\lambda=0.05). Same-field attention remained enabled in all three configurations. We report mean test scores ± sample standard deviations across three independent pretraining seeds.

Auxiliary objectiveMBD-mini Macro-AUROC (%)BPI17 AUROC (%)
None (same-field attention retained)78.66 ± 0.3974.98 ± 0.31
Direct-MSE78.67 ± 0.3674.64 ± 0.30
Paired residual78.53 ± 0.1574.94 ± 0.80

At this fixed weight, the residual objective scored higher than Direct-MSE on BPI17 and lower on MBD-mini. Its relative benefit therefore depends on the dataset.

We will clarify in the revision that the residual auxiliary objective is optional and that SE-JEPA can achieve strong performance with same-field attention alone.

W2 & Q2. Is field alignment necessary for cross-event attention?

Same-field vs. all-field attention. We compared two cross-event attention scopes. Each query could attend either to the corresponding field across events or to all fields across events. Both variants used the same parameter count, initialization, and number of training updates, with the auxiliary objective disabled. The table reports mean test scores ± sample standard deviations (%) across three independent pretraining seeds.

Attention scopeMBD-mini Macro-AUROCBPI17 AUROCAlphaBattle AUROCX5 Accuracy
Same-field78.66 ± 0.3974.98 ± 0.3177.07 ± 0.3555.40 ± 0.02
All-field78.55 ± 0.2874.92 ± 0.3776.50 ± 0.1855.35 ± 0.09

Expanding retrieval to all fields yielded no clear performance gain on the four benchmarks. Same-field attention achieved comparable test scores with lower measured fine-tuning cost. On the same dedicated H200 GPU, using BF16 and a batch size of 32, it reduced fine-tuning time per step by 38.4% on MBD-mini and 19.2% on BPI17 relative to all-field attention.

AuthorsOpenReview ↗

Response to Reviewer 3W8X (Part 2)

W2 & Q3. Clarification on self-retrieval in downstream same-field attention.

During pretraining, we exclude the entire target event from attention memory to prevent retrieval from revealing masked fields. In downstream classification, the query event itself is a valid input. To test the role of self-retrieval, we compared All (self-retrieval allowed) with No-self (self-retrieval disabled throughout fine-tuning and evaluation). Within each configuration, both variants started from the same pretrained weights and used three matched fine-tuning seeds.

Dataset and configurationAllNo-self
BPI17 / Same-field74.36 ± 0.1674.42 ± 0.34
BPI17 / Joint75.03 ± 0.3575.36 ± 0.18
MBD-mini / Same-field79.00 ± 0.0778.64 ± 0.05

Scores are test means ± sample standard deviations (%). Disabling self-retrieval slightly increased mean AUROC for both BPI17 configurations and reduced mean macro-AUROC by 0.36 percentage points on MBD-mini. All three configurations retained strong performance without self-retrieval, although its effect varied by dataset.

We separately analyzed nine All checkpoints on the validation set, with three independently pretrained and fine-tuned models per configuration. The effective gate magnitude ∣tanh⁡(γsf)∣|\tanh(\gamma_{\mathrm{sf}})| was nonzero in all nine (0.136–0.186). For sequences of length 9–16 only, the attention weight assigned to the query event averaged 9.23%–9.45% across configurations after averaging over heads, fields, and queries within each configuration. This was close to the corresponding mean uniform-attention reference (1/L1/L, 8.63%–8.98%), indicating no strong average concentration on the query event within this length group.

Thank you again for your thoughtful feedback. We would be happy to address any further questions.

AuthorsOpenReview ↗

Looking forward to further discussion

Dear Reviewer 3W8X,

Thank you for your thoughtful review and for your positive assessment of our work. We have responded point by point to the concerns and questions you raised. As the discussion deadline approaches, we would appreciate hearing whether the additional experiments and clarifications address your concerns.

  • [Part 1 of our response (W1, W2, Q1, and Q2)]: We conducted controlled comparisons and ablation studies to assess the contribution of individual components. We also added comparisons with relevant baselines you identified.
  • [Part 2 of our response (W2 and Q3)]: We evaluated the effect of excluding the query event itself from the attention memory during downstream fine-tuning and evaluation. We also report the learned gate values and attention-weight statistics.

Thank you again for your valuable suggestions.

Best regards,

37131 Authors

Reviewer_28jpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SE-JEPA (Structured-Event Joint-Embedding Predictive Architecture), a self-supervised representation learning framework specifically designed for structured event sequences (such as financial transactions, retail purchases, and business process logs).

  • Core Motivation: Structured event data contains both "intra-event field dependencies" and "inter-event contextual relationships." Traditional methods have limitations in handling heterogeneous fields (a mixture of numerical and categorical types) and organizing prediction targets. SE-JEPA combines latent-space prediction with multi-granularity cross-event context:
  1. Same-field cross-event attention: In the main prediction path, field-level representations are retrieved from historical events by aligning identical fields.
  2. Event-level residual auxiliary learning: Utilizes pooled event representations to construct an auxiliary branch, allowing the context adjustment to fit local prediction errors via residual targets.
  3. Teacher-student architecture: Employs an Exponential Moving Average (EMA) teacher network to provide stop-gradient latent-space targets.
  • Experimental Performance: Across four public benchmark datasets (MBD-mini, AlphaBattle, Retail/X5, and BP117), SE-JEPA achieves leading or strong performance in the majority of tasks under both frozen linear evaluation and full fine-tuning.
Critical strengths
  • Multi-Granularity Decoupled Modeling: Successfully decouples and integrates "microscopic field-level interactions" and "macroscopic inter-event historical context" architecturally, preserving fine-grained features while incorporating long-range behavioral patterns.
  • Inductive Bias: Employs "same-field alignment attention" to force the model to search for corresponding similar fields in historical events (e.g., amount-to-amount), which aligns better with the business logic of structured data than blindly applying mixed attention across all fields.
  • Experiments and Generalization: Comprehensively validated across four cross-domain datasets (finance, retail, business logs), demonstrating exceptional label efficiency in low-label data scenarios (20%–80% labels).
  • Mathematical Formulation: The design of Event-level Residual Auxiliary Learning ensures that local predictions are not disrupted at initialization, while allowing gradients to perceive the Jacobian changes brought by the context.
Critical weaknesses
  • "Engineering Heuristics" Architectural Design: The model introduces numerous components and switches (such as the same-field switch aa, auxiliary loss weight λ\lambda, and scaling factors γ\gamma with zero-initialization), leading to high costs and complexity in hyperparameter tuning.
  • Strong Dependency on Fixed Schemas: Current formulations and same-field alignments heavily rely on all events having a fixed and consistent field definition (Schema). Scalability is limited when facing real-world industrial scenarios where fields frequently change or schemas are heterogeneous across tables.
  • Computational and Memory Overhead: Due to the need to retrieve cross-event attention for same-field matching for each target field in the main path, along with independent auxiliary pre-training branches, its pre-training and inference computational overhead may increase significantly compared to simple pooling or pure sequence models.
Questions for authors
  1. Generalizability to Schema Changes:
  • Question: The current architecture heavily relies on "same-field cross-event attention," requiring strictly aligned field identities (schemas) across different events. However, in real-world business systems, fields frequently evolve (e.g., adding/deleting attributes, or inconsistent schemas across business lines). How should SE-JEPA adapt when encountering scenarios with dynamically changing schemas?
  1. Marginal Utility of the Auxiliary Loss:
  • Question: Judging from the ablation studies (Table 2), event-level residual auxiliary learning sometimes even leads to performance drops when used alone (e.g., on Retail/X5 and BP117), only showing synergistic effects when combined with same-field attention. Does this imply that the auxiliary loss is extremely sensitive to hyperparameters? Are there more direct and lightweight ways to replace this complex dual-branch residual design?
  1. Computational Cost and Industrial Deployment:
  • Question: Compared to traditional contrastive learning models that compress the entire event sequence into a single vector (e.g., COLES), SE-JEPA needs to maintain complex memory at both the field and event levels and perform high-dimensional attention retrieval. Under real-world industrial massive data and long-sequence scenarios, how does its training and inference latency perform? Have you had to truncate the historical context window due to computational bottlenecks?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 28jp (Part 1)

We sincerely thank you for your thoughtful review and for highlighting both the rationale for same-field alignment in structured event sequences and SE-JEPA’s exceptional label efficiency. Below we address the concerns and questions raised in the review.

W1. Cost and complexity of hyperparameter tuning

Thank you for raising the issue of design complexity and tuning cost. We will distinguish more clearly among module configurations, parameters learned during training, and hyperparameters that require selection.

Both scaling factors are initialized to zero and learned end to end with the model. Their initial values therefore require no dataset-specific search. At initialization, the added contextual offsets are zero, preserving the local model’s predictions while allowing the gates to receive a learning signal. The same-field switch specifies the module configuration. The auxiliary loss weight λ\lambda is a hyperparameter that must be set. For the revised main table, we enable same-field retrieval and fix λ = 0.05 on all four datasets.

To examine the effect of λ\lambda in isolation, we kept same-field retrieval enabled and compared four positive weights with a reference configuration in which the auxiliary loss was disabled. The table reports HPO-Val scores under the original evaluation protocol, expressed as the mean ±\pm sample standard deviation across two development seeds. These validation scores are used solely to compare the weights. The main table reports test-set results.

Auxiliary weight (λ)MBD-mini Macro-AUROC (%)BPI17 AUROC (%)
0 (auxiliary loss disabled)80.34 ± 0.3275.75 ± 0.50
0.0180.51 ± 0.3776.20 ± 0.23
0.02580.40 ± 0.3475.55 ± 0.22
0.0580.56 ± 0.2275.57 ± 0.19
0.1080.51 ± 0.4476.06 ± 0.07

Across the four positive weights, the mean validation scores span 0.16 percentage points on MBD-mini and 0.64 percentage points on BPI17. These results indicate that sensitivity to λ\lambda varies by dataset.

W2 & Q1. Dependence on a fixed schema and generalization under schema changes

Thank you for raising this issue. As described in Section 3.1 and Appendix A.2 of the manuscript, SE-JEPA preserves field positions and explicitly encodes naturally missing values in known fields using learned, type-specific missing-value representations and observation-status embeddings.

We distinguish three types of field changes in industrial systems.

  1. A known field is missing or disabled. Its original position is retained, and unavailable values are marked as missing. This case is covered by the missing-value encoding described in the manuscript and by the perturbation experiments on four datasets reported below.

  2. A field is renamed and its correspondence is known. An external input adapter can map the renamed field to its existing field ID, provided that the field semantics, units, and categorical encodings are compatible.

  3. A field is new or cannot be reliably aligned with an existing field. The model can initially use the existing fields that can be aligned. Using a new attribute requires extending the field representations and training the adapted model on new data.

Transfer across business lines also requires checking that event and time definitions are compatible and evaluating performance under distribution shift.

To simulate changes in field availability in industrial systems, we evaluated the Same-field configuration under two perturbations: random masking of individual field values and persistent unavailability of entire fields. We held the model weights fixed throughout evaluation. For MBD-mini, BPI17, and AlphaBattle, disabling approximately 10% or 30% of the field groups corresponded to disabling one or four groups, respectively. For X5, the corresponding counts were two and five.

Change in validation performance (percentage points)

PerturbationMBD-miniBPI17AlphaBattleX5
Randomly mask 10% of field values−1.69±0.17-1.69 \pm 0.17−4.11±0.39-4.11 \pm 0.39−1.58±0.12-1.58 \pm 0.12−1.30±0.05-1.30 \pm 0.05
Disable approximately 10% of field groups−0.95±0.34-0.95 \pm 0.34−1.55±1.41-1.55 \pm 1.41−2.63±3.50-2.63 \pm 3.50−3.10±5.56-3.10 \pm 5.56
Randomly mask 30% of field values−4.06±0.16-4.06 \pm 0.16−10.23±0.50-10.23 \pm 0.50−4.12±0.21-4.12 \pm 0.21−7.57±0.17-7.57 \pm 0.17
Disable approximately 30% of field groups−2.92±1.51-2.92 \pm 1.51−13.15±7.91-13.15 \pm 7.91−6.71±2.67-6.71 \pm 2.67−12.83±11.35-12.83 \pm 11.35

Disabling one known field group reduced validation performance by 0.95 percentage points on MBD-mini and 1.55 percentage points on BPI17. When four field groups were disabled, the corresponding drops increased to 2.92 and 13.15 percentage points, respectively. These results quantify the model’s sensitivity to missing values in known input fields.

AuthorsOpenReview ↗

Response to Reviewer 28jp (Part 2)

W3. Computational and GPU memory overhead.

Thank you for raising this issue. We compared SE-JEPA (same-field) with five baselines on MBD-mini using the same hardware and input batches. Test scores are means ± sample standard deviations over five runs. Latency is the median forward-pass time for a batch of 32 sequences.

MethodMacro-AUROC (%) ↑Inference latency (ms/batch) ↓Peak allocated GPU memory (MiB) ↓
GRU (supervised)75.24 ± 0.543.76054.15
Transformer (supervised)77.41 ± 0.631.55254.35
CoLES75.10 ± 0.483.78354.15
T-JEPA (mean readout)74.81 ± 0.161.96872.25
TabBERT77.82 ± 0.122.252180.45
SE-JEPA (same-field)78.66 ± 0.283.437150.17

Inference. SE-JEPA achieved the highest mean macro-AUROC among the six methods. Compared with GRU and CoLES, it scored 3.42 and 3.56 percentage points higher with 8.6% and 9.1% lower batch latency, respectively, at the cost of higher peak memory. Compared with TabBERT, it gained 0.84 percentage points and used 16.8% less peak GPU memory, taking an additional 1.185 ms per batch.

Training. On MBD-mini, adding the event-level auxiliary objective increased pretraining time per step by 32.5% and peak GPU memory by 30.6% (298.0 to 389.3 MiB). Only same-field retrieval remains active at inference. In a separate comparison, same-field and all-field retrieval yielded similar mean test scores, while same-field retrieval reduced fine-tuning time per step by 38.4% on MBD-mini and 19.2% on BPI17. We will include the full timing and memory measurements in the revised appendix.

Q2. Marginal utility, weight sensitivity, and alternatives to the auxiliary loss

Thank you for this question. The residual objective uses event-level context to correct the local prediction. Direct-MSE instead fits the contextual prediction directly to the teacher target.

Marginal benefit. Over five independent pretraining and fine-tuning runs, adding the residual objective (λ = 0.05) raised mean BPI17 test AUROC by over 0.33 percentage points both without same-field attention (Aux-only vs. Local) and with it (Joint vs. Same-field). On Retail/X5, mean accuracy was nearly unchanged in the first comparison and fell by 0.10 points in the second. The mean gain from the auxiliary objective varies across datasets.

The residual auxiliary objective is an optional component of SE-JEPA. It uses event-level context to correct local predictions, and its use can be guided by validation performance. As described in Section 3.3, Eq. (8), setting λ=0\lambda=0 disables the auxiliary branch and avoids its pretraining cost while retaining same-field attention.

Weight sensitivity. Across λ = 0.01, 0.025, 0.05, and 0.10, mean validation scores over two development seeds varied by 0.64 points on BPI17 and 0.16 on MBD-mini. The revised main table uses λ = 0.05 on all four datasets.

Alternatives. With same-field attention enabled, we selected each objective’s weight from the same validation grid. Test scores are means ± sample standard deviations over three paired pretraining and fine-tuning runs.

Dataset and metricNo auxiliary lossDirect-MSEPaired residual
BPI17 AUROC (%)74.98 ± 0.3174.49 ± 0.2774.79 ± 0.28
MBD-mini macro-AUROC (%)78.66 ± 0.3978.74 ± 0.3978.53 ± 0.15

The residual objective outperformed Direct-MSE by 0.31 points on BPI17, while Direct-MSE led by 0.21 points on MBD-mini. For a lighter configuration, setting λ = 0 retains same-field attention and removes the auxiliary branch (Section 3.3, Eq. 8), reducing MBD-mini pretraining time from 76.90 to 58.04 ms per step.

We will distinguish the residual objective’s rationale from its observed gains in the revision, without implying that it is essential for strong performance.

AuthorsOpenReview ↗

Response to Reviewer 28jp (Part 3)

Q3. Long histories, truncation, and deployment cost

Performance and latency. SE-JEPA retrieves same-field context only within the input sequence. We compared Joint and CoLES on the same H200 GPU using identical window limits. At batch size 32, Joint achieved 3.48 percentage points higher mean test macro-AUROC on MBD-mini with similar local inference latency (5.19 vs. 5.40 ms/batch). On BPI17, Joint achieved 4.14 percentage points higher mean test AUROC at an additional 1.10 ms/batch. Joint had higher pretraining time per step and higher single-sequence inference latency on both datasets.

Truncation and window length. The 140-event window covered every BPI17 training history. On MBD-mini, retaining the 150 most recent events within the reporting month truncated 0.37% of training histories and discarded 1.63% of training events. With three Same-field checkpoints held fixed, shortening the window to 64 events reduced mean validation AUROC by 0.03 percentage points on BPI17 and macro-AUROC by 0.01 percentage points on MBD-mini. On a matched batch of 32 long histories per dataset, forward-pass latency fell by 15.9% and 14.8%, respectively. Applications that rely on longer histories should verify accuracy and serving latency on those histories.

Timing (ms)BPI17 JointBPI17 CoLESMBD-mini JointMBD-mini CoLES
Pretraining step, batch 3270.7310.7076.9015.79
Local inference, batch 13.712.163.731.95
Local inference, batch 325.244.145.195.40
Forward pass only, batch 13.631.833.601.62
Forward pass only, batch 323.822.483.873.82

Local inference includes CPU batching, host-to-device transfer, and the forward pass.

Dataset and validation metricOriginal window (%)128 events (%)64 events (%)Forward latency, original → 64 events (ms/batch)
BPI17 AUROC75.932275.931075.90264.60 → 3.87
MBD-mini macro-AUROC80.589380.590680.57894.03 → 3.43

Validation scores are averaged over three checkpoints on each full validation set. Both latency measurements use the same long-history batch within each dataset.

Thank you again for your thoughtful review and constructive feedback. We would be happy to address any further questions.

AuthorsOpenReview ↗

Looking forward to further discussion

Dear Reviewer 28jp,

Thank you for your thoughtful review of our submission. We have responded point by point to the concerns and questions you raised. As the discussion deadline approaches, we would appreciate hearing whether the additional experiments and clarifications address your concerns.

  • [Part 1 of our response (W1, W2, and Q1)] We examined sensitivity to the auxiliary loss weight and measured the effects of missing field values and of entire groups of known fields becoming unavailable.
  • [Part 2 of our response (W3 and Q2)] We report measured training and inference costs and GPU memory usage. We also assess the incremental benefit of the residual auxiliary objective and compare it with Direct-MSE as an alternative.
  • [Part 3 of our response (Q3)] We evaluated how shortening the event history window affects validation performance and inference latency.

Thank you again for your valuable suggestions.

Best regards,
37131 Authors

Comments on the paper

1 comment

Dear ACs and Reviewers,

We sincerely thank all reviewers for taking valuable time to provide profound and constructive comments on our work.

During the rebuttal period, we responded point by point to the comments from all three reviewers and conducted extensive additional experiments in response. We have also uploaded an updated revised manuscript. For the convenience of the ACs, we will summarize each reviewer’s comments along with our responses.

Key Strengths Highlighted by Reviewers

  • Important problem and clear motivation. GaEX recognized the importance of structured-event representation learning and noted that the problem remains underexplored. 3W8X highlighted the clear motivation for combining within-event field dependencies with cross-event context.
  • Field-aligned contextual modeling. 28jp and 3W8X recognized the rationale for same-field attention and for exploiting the semantic structure of event fields. 28jp also highlighted the integration of fine-grained field interactions with event-level historical context.
  • Complete JEPA formulation and training design. GaEX recognized the value of studying a complete JEPA pipeline for structured event sequences and the rationale for zero-initialized gates and two-stage training. 28jp highlighted the residual formulation’s preservation of local predictions at initialization.
  • Evaluation coverage and label efficiency. 28jp and 3W8X highlighted the evaluation across four benchmarks and multiple settings, including full fine-tuning, frozen evaluation, label efficiency, and component ablations. 28jp particularly noted the performance under limited-label conditions.

Reviewers’ Concerns and Our Responses

Reviewer 28jp

  • [Part 1 of our response (W1, W2, and Q1)]: We examined sensitivity to the auxiliary loss weight and measured the effects of missing field values and of entire groups of known fields becoming unavailable.
  • [Part 2 of our response (W3 and Q2)]: We report measured training and inference costs and GPU memory usage. We also assess the incremental benefit of the residual auxiliary objective and compare it with Direct-MSE as an alternative.
  • [Part 3 of our response (Q3)]: We evaluated how shortening the event history window affects validation performance and inference latency.

Reviewer 3W8X

  • [Part 1 of our response (W1, W2, Q1, and Q2)]: We conducted controlled comparisons and ablation studies to assess the contribution of individual components. We also added comparisons with relevant baselines you identified.
  • [Part 2 of our response (W2 and Q3)]: We evaluated the effect of excluding the query event itself from the attention memory during downstream fine-tuning and evaluation. We also report the learned gate values and attention-weight statistics.

Reviewer GaEX

  • [Part 1 of our response (W1)]: We investigated performance gains using matched readouts, starting checkpoints, and continuation updates, and compared retaining versus bypassing the pretrained same-field module downstream.
  • [Part 2 of our response (W2, Q1)]: We clarified the relationship between the main and ablation tables, reported five-run means and sample standard deviations, and specified configuration and repetition protocols.
  • [Part 3 of our response (W3, W4, Q5)]: We reported EBES reruns, described limited-budget validation search, investigated the CoLES frozen probe, and clarified the relationship to the closest prior work.
  • [Part 4 of our response (Q2, Q3)]. We detailed the sampling, masking, readout, and budget differences between Local-only and T-JEPA, added readout and encoder-transfer comparisons, and documented pretraining settings.
  • [Part 5 of our response (Q4)]: We compared latent and observation-space targets, predictor-side and encoder-side attention, and same-field and all-field retrieval under controlled settings.

Discussion Status

Reviewer GaEX acknowledged that the additional controls and experiments addressed many of the original concerns, requested further clarification of some reported results and evaluation protocols, and expressed openness to reconsidering the rating following clarification. We have posted a follow-up response addressing these questions individually and invited all three reviewers to comment on the additional evidence.

We sincerely thank all reviewers and ACs for their time, careful assessments, and constructive feedback throughout the discussion.

Best regards,
All Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.