acceptodds
Under review as a conference paper at ICLR 2027

Learning Predictive Event States for Sequential Modeling and Recommendation

Abstract

Next-token prediction (NTP) trains causal sequence models for sequential recommendation, but its next-interaction target does not specify which event-level information should be accessible from each causal state. We introduce Event State Prediction (ESP), which augments NTP with two predictive relations over contiguous interaction groups. Intra-event prediction trains token states formed from observed prefixes to predict contextual summaries of their enclosing events; inter-event prediction trains pooled states at event completion to predict the next event before its interactions are observed. Both relations use history-conditioned targets from an exponential-moving-average encoder and single-positive contrastive matching, supervising individual states and event-level aggregates. ESP improves NDCG@10 and NDCG@20 across four datasets while retaining the original NTP inference path. Frozen-state probes on three datasets indicate greater accessibility of future and event-level information on the tested tasks. Geometry analyses find higher effective rank and broader state dispersion, and ablations support the complementary contributions of the two objectives. The code is available at https://anonymous.4open.science/r/ESP-Code/.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Generative Recommendation, Sequential Modeling, Predictive Representation Learning, Event State Prediction

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_c4uvOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Event State Prediction (ESP) for sequential recommendation. Specifically, the method adds two event-level objectives to standard next-token prediction. One focuses on learning representations within an event, while the other predicts the next event from the current event state. Both use contrastive learning with an EMA encoder. Experiments are conducted on four datasets using the HSTU backbone, with comparisons against recent methods.

Critical strengths
  1. The proposed objectives introduce event-level supervision without changing the backbone or increasing inference-time model complexity.
  2. The experimental design is relatively sufficient.
Critical weaknesses
  1. As noted in Section 4.2, later causal teacher states can access this token even after leave-one-out pooling. It seems that the intra-event target may still contain information from the source token.
  2. ESP takes 1.37× longer to train than NTP on MerRec, but no comparison under the same computational budget is provided. The additional training cost may need further consideration.
  3. The performance gains of ESP on KuaiRec are relatively small. However, Appendix D.3 indicates that the TDL component of NONTP is disabled on this dataset. It is unclear how this modification affects the comparison, which may need further clarification.
Questions for authors
  1. Could the authors clarify how much information from the source token remains in the intra-event target after leave-one-out pooling?
  2. Could the authors compare ESP with NTP under the same computational budget?
  3. Could the authors explain why the TDL component of NONTP is disabled on KuaiRec?
3 comments

Source-Token Information in Intra-Event Targets (W1, Q1)

W1: As noted in Section 4.2, later causal teacher states can access this token even after leave-one-out pooling. It seems that the intra-event target may still contain information from the source token.

Q1: Could the authors clarify how much information from the source token remains in the intra-event target after leave-one-out pooling?

We thank the reviewer for highlighting this subtle point. Leave-one-out pooling excludes the source position's teacher state, but subsequent causal teacher states may still contain information from the source token. Importantly, this does not introduce future information into the causal online query.

To quantify this residual dependence, we measure target sensitivity using fixed-slot source exclusion: we zero the source input embedding and its keys and values at each attention layer while keeping sequence positions, timestamps, and event membership fixed. Across MerRec, RetailRocket, and KuaiRec, target cosine similarities for First/Middle/Last positions are 0.9976/0.9995/1.0000, 0.9839/0.9938/1.0000, and 0.9923/0.9981/1.0000, respectively. This confirms measurable but decreasing sensitivity toward event boundaries.

To assess the impact of this residual information, we conduct additional target-construction ablations, including strict source deletion, which removes the source token from the teacher input and recomputes the event target (NDCG@10/HR@10).

MethodMerRecRetailRocketKuaiRec
ESP0.2307/0.38030.1520/0.18530.1177/0.1587
Sampled LOO0.2291/0.37830.1523/0.18660.1175/0.1583
Strict source deletion0.2311/0.37800.1525/0.18650.1177/0.1585

While target similarity does not directly measure mutual information, these results indicate that ESP's gains do not critically depend on residual source-token information in the intra-event targets.

Training Efficiency and Compute-Matched Comparison (W2, Q2)

W2: ESP takes 1.37× longer to train than NTP on MerRec, but no comparison under the same computational budget is provided. The additional training cost may need further consideration.

Q2: Could the authors compare ESP with NTP under the same computational budget?

To assess whether ESP's gains simply arise from additional training computation, we extend NTP training to 14 epochs, providing comparable or larger estimated training-time budgets than ESP across the three datasets (NDCG@10/HR@10).

MethodMerRecRetailRocketKuaiRec
NTP (7 epoch)0.1759/0.25890.1099/0.14640.1149/0.1548
NTP (10 epoch)0.2048/0.31870.1200/0.15770.1163/0.1572
NTP (14 epoch)0.2074/0.31840.1203/0.15760.1160/0.1556
ESP0.2307/0.38030.1520/0.18530.1177/0.1587

ESP consistently outperforms the longer-trained NTP baseline, indicating that its gains cannot be explained merely by additional training time. Furthermore, Appendix F.4 (Figure 14) shows that ESP achieves a lower late-stage NTP training loss. Importantly, ESP introduces no additional inference cost, as all auxiliary components are discarded after training.

NONTP Configuration and KuaiRec Performance (W3, Q3)

W3: ... However, Appendix D.3 indicates that the TDL component of NONTP is disabled on this dataset. It is unclear how this modification affects the comparison, which may need further clarification.

Q3: Could the authors explain why the TDL component of NONTP is disabled on KuaiRec?

We thank the reviewer for requesting this clarification. As described in Appendix D.3, NONTP's TDL requires behavioral domain labels, using historical states from domains different from that of the next prediction target. For MerRec, RetailRocket, and AppRec, we use action types as domain mappings and retain the full NONTP objective (TCL + TDL). In contrast, KuaiRec consists of video interactions without a corresponding action-domain label. We therefore retain TCL while disabling the inapplicable TDL component, rather than introducing arbitrary domain assignments.

This is a dataset-specific adaptation, not a selective modification of the baseline.

We have revised the manuscript, and highlight the main changes in green. The full setup for above experiments are provided in Appendix E.2.3 and F.5.2. We hope these additions address your concerns, and we appreciate your valuable feedback.

I appreciate the authors' careful responses and the additional analyses in the revised manuscript. The source-exclusion study and strict source-deletion ablation provide useful evidence that the residual source-token dependence in the intra-event targets does not materially affect the reported gains. The compute-matched comparison with longer-trained NTP baselines also strengthens the experimental evaluation. Moreover, the explanation for using a TCL-only NONTP variant on KuaiRec is reasonable given the absence of suitable action-domain labels. Overall, I find that the authors have adequately addressed my main concerns, and the revision has improved the clarity and credibility of the work. I am therefore inclined to increase my rating from 3 to 4.

Thank you! We are happy that our rebuttal addressed your concerns, and we thank you again for your constructive suggestions.

Reviewer_6P57OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ESP (Event State Prediction), an auxiliary loss built on top of next‑token prediction (NTP) for sequential recommendation. Given pre‑defined event partitions over interaction sequences, it proposes intra‑event and inter‑event contrastive objectives with EMA‑based event targets. All auxiliary components are discarded at inference so the runtime matches vanilla NTP. The authors conduct extensive experiments across four datasets, including ranking metrics, representation geometry diagnostics, ablation studies, and hyper‑parameter sensitivity analysis.

Critical strengths
  1. The inference path is identical to standard NTP; no extra computational cost for online serving.
  2. Empirical evaluation is heavy: multiple datasets, probing analysis, geometry metrics, ablations, hyper‑parameter sweeps.
  3. The distinction between token‑level supervision and event‑level supervision is conceptually clear.
Critical weaknesses
  1. The whole framework hinges on externally provided event boundaries, mostly fixed time windows in the paper. Performance fluctuates substantially with different partition granularities, as shown in the appendix. The authors do not propose any learnable partition mechanism. In many real‑world recommendation scenarios, clean time‑based event boundaries simply do not exist. Tuning partition granularity via validation set creates an additional heavy hyper‑parameter search burden. Without a robust default partitioning strategy, the practical applicability of ESP is questionable. 2.The core ingredients are all existing techniques, the main contribution is combining them into intra‑/inter‑event training losses for sequential recommendation. There is no new model architecture, no new contrastive objective formulation. Event/session‑aware self‑supervision has long been studied in sequential recommender systems.
Questions for authors
  1. Given that event partition heavily impacts performance, what is your out‑of‑the‑box solution for datasets without natural time boundaries? Can ESP work robustly across diverse partition choices without validation‑set tuning?
  2. How do you disentangle whether performance gains come from event grouping itself or from the InfoNCE contrastive matching objective? Could you provide results for a variant with event‑pooling auxiliary loss but no contrastive discrimination?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 6P57 (Part 1/2)

Partition Robustness and Practical Applicability (W1, Q1)

W1: The whole framework hinges on externally provided event boundaries, mostly fixed time windows in the paper. Performance fluctuates substantially with different partition granularities, as shown in the appendix. The authors do not propose any learnable partition mechanism. In many real‑world recommendation scenarios, clean time‑based event boundaries simply do not exist. Tuning partition granularity via validation set creates an additional heavy hyper‑parameter search burden. Without a robust default partitioning strategy, the practical applicability of ESP is questionable.

Q1: Given that event partition heavily impacts performance, what is your out‑of‑the‑box solution for datasets without natural time boundaries? Can ESP work robustly across diverse partition choices without validation‑set tuning?

We thank the reviewer for raising this concern.

ESP does not require annotated session boundaries or learnable partition mechanisms. We adopt deterministic 1-hour windows when timestamps are available and Fix-5 (five consecutive interactions per event) as an out-of-the-box alternative otherwise. Neither strategy requires dataset-specific partition tuning.

Based on your suggestion, we evaluate fixed-time, fixed-count and inactivity-based sessions (NDCG@10/HR@10).

PartitionMerRecRetailRocketKuaiRec
ESP (1h)0.2307/0.38030.1520/0.18530.1177/0.1587
0.5h0.2291/0.37840.1515/0.18480.1168/0.1577
2h0.2286/0.37900.1519/0.18550.1172/0.1586
Fix-50.2274/0.37840.1511/0.18410.1174/0.1584
Session-30m0.2286/0.37930.1515/0.18460.1176/0.1587

These results show that ESP is robust across a range of reasonable local partition choices, while overly coarse partitions can still degrade performance (mentioned in Appendix F3.2). Moreover, a simple, consistently applied fixed-count rule provides a practical, tuning-free alternative when natural session boundaries are unavailable.

Novelty Beyond Existing Components (W2)

2.The core ingredients are all existing techniques, the main contribution is combining them into intra‑/inter‑event training losses for sequential recommendation. There is no new model architecture, no new contrastive objective formulation. Event/session‑aware self‑supervision has long been studied in sequential recommender systems.

We would kindly argue that the key contribution of ESP is the formulation and joint supervision of two complementary event-relational predictive requirements, distinguished by their source granularity and prediction scope.

  • Intra-event prediction requires individual causal token states to predict contextual summaries of their enclosing events, including future event content when the source lies within an unfolding event.

  • Inter-event prediction requires a completed event's pooled state to predict the subsequent event before any of its interactions are observed.

ESP differs from the closest related methods in the following ways:

  • NONTP predicts future token states at fixed offsets, rather than explicitly supervising event-level source–target relations.
  • S^3-Rec employs segment-level self-supervision but does not jointly impose the above token-to-enclosing-event and completed-event-to-next-event predictive requirements.
  • SessionRec predicts next-session positive items from session representations, without directly supervising individual causal token states against their enclosing-event representations.

ESP's key distinction is not merely what to predict, but which causal representation is required to make the prediction. Its intra-event and inter-event objectives impose complementary constraints on individual token states and completed-event representations, as formally analyzed in Appendix B.3 and empirically supported by the ablations and frozen-state analyses (Secs. 5.2–5.6).

AuthorsOpenReview ↗

Response to Reviewer 6P57 (Part 2/2)

Event Grouping vs. Contrastive Matching (Q2)

How do you disentangle whether performance gains come from event grouping itself or from the InfoNCE contrastive matching objective? Could you provide results for a variant with event‑pooling auxiliary loss but no contrastive discrimination?

Thank you for your careful reading and comments. We would clarify that the non-contrastive control experiment in Table 3 and Appendix E.2 shows that the cosine regression variant retains the same event pooling and predictive assignments while replacing InfoNCE with a non-contrastive (cosine) loss. We further include token-level contrastive baselines to disentangle the contributions of event-level supervision and contrastive matching.

MethodNDCG@10NDCG@20HR@10MRRRankMe/dIsoScore
NextLat0.17690.19190.27770.18130.61330.1701
NextLat + InfoNCE0.20100.21700.34650.19070.70350.2258
NTP + Token-level InfoNCE0.20180.21710.30310.20430.70460.2286
ESP + Cosine0.20330.21790.33450.19490.72140.2123
ESP0.23070.24590.38030.21400.73160.2443

Neither event pooling with cosine regression nor token-level contrastive learning alone reproduces ESP's gains. These results highlight the importance of combining event-aligned predictive supervision with contrastive discrimination, rather than attributing the improvements to either component alone.

We have revised the manuscript, and highlight the main changes in blue. The full setup for above experiments are provided in Section 5.8 (marked in red) and Appendix E.2.2. We hope these additions address your concerns, and we appreciate your valuable feedback.

I appreciate the authors' detailed response, which clarified my misunderstanding regarding event partitioning. Furthermore, the additional results demonstrate that ESP is robust across a range of reasonable local partition choices. The authors also provided a satisfactory explanation concerning novelty; in light of this, I am raising my score to 3.

Thank you! We sincerely appreciate your prompt feedback and the time you took to consider our rebuttal.

Reviewer_hiJbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Existing recommendation models primarily rely on next-token prediction (NTP) as their training objective. This paper proposes Event State Prediction (ESP), which introduces two additional objectives, intra-event prediction and inter-event prediction, to encourage learned representations to capture richer information about future user behaviors. Events are constructed using fixed one-hour time windows, and the proposed framework leverages a teacher-student architecture with knowledge distillation to incorporate event-level predictive signals.

Critical strengths

S1. The paper clearly identifies the limitations of next-token prediction supervision and provides a well-motivated and convincing explanation of why event-level prediction is important.

S2. The ablation study shows a substantial performance difference depending on whether intra-event prediction is included, highlighting the importance of this objective.

Critical weaknesses

W1. Since the proposed method relies on event-level prediction, I believe it is important to understand how different event partitioning strategies affect its performance. I acknowledge that the results presented in the appendix demonstrate that the proposed one-hour partition achieves the best performance among the evaluated settings. However, beyond these empirical results, I find the justification for constructing non-overlapping events based on fixed time intervals somewhat insufficient. In practice, temporal dependencies in user behavior may not necessarily align with fixed time boundaries. Therefore, I believe the paper would benefit from further analysis or a more detailed rationale explaining how the design of event partitions affects the proposed methodology, beyond simply comparing their empirical performance.

Questions for authors

Q1. The proposed methodology appears to be largely architecture-agnostic. Have the authors considered applying it to different backbone architectures?

Q2. As mentioned in W1, I would be interested in hearing the authors’ perspectives on the choice of event partitioning strategy, particularly the use of fixed, non-overlapping time windows.

3 comments

Event Partition Design (W1, Q2)

W1: ... However, beyond these empirical results, I find the justification for constructing non-overlapping events based on fixed time intervals somewhat insufficient. In practice, temporal dependencies in user behavior may not necessarily align with fixed time boundaries. Therefore, I believe the paper would benefit from further analysis or a more detailed rationale explaining how the design of event partitions affects the proposed methodology, beyond simply comparing their empirical performance.

Q2: As mentioned in W1, I would be interested in hearing the authors’ perspectives on the choice of event partitioning strategy, particularly the use of fixed, non-overlapping time windows.

We thank the reviewer for this insightful question. Event boundaries in ESP are not assumed to correspond to ground-truth behavioral intents. Instead, they define a local predictive supervision structure. Short partitions provide contextual targets containing multiple interactions, while excessively long partitions may aggregate heterogeneous behavioral contexts and weaken the locality of supervision (Appendix F.3). Importantly, the partition does not constrain causal attention.

We use non-overlapping 1-hour windows as a simple, reproducible default, yielding roughly 4–6 interactions per event. This also ensures that each interaction is assigned to one event.

Based on your suggestion, we further evaluate fixed-time windows, fixed-count grouping, inactivity-based sessions, and shifted boundaries (NDCG@10/HR@10).

PartitionMerRecRetailRocketKuaiRec
ESP (1h)0.2307/0.38030.1520/0.18530.1177/0.1587
0.5h0.2291/0.37840.1515/0.18480.1168/0.1577
2h0.2286/0.37900.1519/0.18550.1172/0.1586
Fix-50.2274/0.37840.1511/0.18410.1174/0.1584
Session-30m0.2286/0.37930.1515/0.18460.1176/0.1587
1h(30m shift)0.2292/0.37850.1519/0.18530.1173/0.1585

These results demonstrate that ESP does not require precisely aligned event boundaries.

Backbone Generalization (Q1)

Q1. The proposed methodology appears to be largely architecture-agnostic. Have the authors considered applying it to different backbone architectures?

Thanks for your insightful comment. ESP is an architecture-agnostic training objective operating on causal states, rather than an HSTU-specific modification. The original manuscript demonstrates its effectiveness with SASRec on MerRec, improving recommendation metrics. We further extend the evaluation to SASRec and Ultra-HSTU on three datasets (NDCG@10/HR@10).

MethodMerRecRetailRocketKuaiRec
SASRec+NTP0.2014/0.32120.1162/0.15310.1203/0.1644
SASRec+ESP0.2228/0.3685(+11%/+15%)0.1296/0.1821(+12%/+19%)0.1229/0.1715(+2.2%/+4.3%)
Ultra-HSTU+NTP0.2280/0.35700.1169/0.15840.1183/0.1645
Ultra-HSTU+ESP0.2570/0.3947(+13%/+11%)0.1368/0.1824(+17%/+15%)0.1216/0.1692(+2.8%/+2.9%)

ESP consistently improves both NDCG@10 and HR@10 across all six backbone–dataset combinations, demonstrating that its benefits generalize beyond the original HSTU backbone.

We have revised the manuscript, and highlight the main changes in red. The full setup for above experiments are provided in Sections 5.8 and 5.7. We hope these additions address your concerns, and we appreciate your valuable feedback.

Thank you for the rebuttal. My concerns have been adequately addressed, and I will raise my score from 2 to 3.

Thank you! We are happy that our rebuttal addressed your concerns and sincerely appreciate your constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.