acceptodds
Under review as a conference paper at ICLR 2027

Beyond Routine: A Controlled Benchmark of Event Grounding in Next-POI Prediction

Abstract

Next-point-of-interest (next-POI) prediction forecasts a user’s next destination from their observed visit history. Events provide additional context, but predictive event utility does not necessarily imply event grounding. Context-on/off evaluation can reveal a gain without establishing whether it depends on the supported temporal and spatial bindings. We introduce BEYONDROUTINE, a controlled benchmark for event grounding in next-POI prediction under an event-aware input setting. Its construction combines chronological mobility records with independently documented events and requires evidence that event information was available before prediction. Events are matched to prediction instances without consulting the future target. Each comparison fixes the prediction instance, candidates, target, and trained model while removing event information or changing its time or location. We further introduce EventResidual, a reference predictor that factorizes history and event representations and applies gated, candidate-specific score corrections over shared POI representations. In the event-conditioned model study, the evaluated predictors show different utility and binding profiles, while stricter controls reveal additional variation in binding sensitivity. EventResidual shows positive paired point estimates on all three primary evaluation sets, including an event-utility difference of 0.0233 in MRR@10.

Primary area:
datasets and benchmarks
Keywords:
human mobility, next-POI prediction, event grounding, mobility benchmark, spatiotemporal modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Mrt2OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

This paper investigates the impact of urban events on the prediction of the next POI, focusing on whether models---while leveraging event information to improve prediction performance---actually utilize the specific temporal and spatial details of those events. To this end, the authors construct a benchmark called BEYOND ROUTINE, which links check-in trajectories with independently collected event records. Using this benchmark, they evaluate the utility of event information and the extent to which models rely on temporal and spatial data by systematically removing events, altering their timing, or replacing their locations within the prediction samples. Furthermore, the authors propose a reference model, EventResidual, which combines scoring based on historical trajectories with a gating mechanism for event-based adjustments, and conduct comparative experiments against modified baseline models using this benchmark.

Critical strengths

The Related Work section is very clearly structured. It provides a well-organized overview, ranging from datasets on human mobility to the classification of methods for next-location prediction, and the inclusion of a comparative table helps readers quickly grasp the progress made in this field.

Critical weaknesses
  1. The introduction and the problem definition section are not reader-friendly, with concepts introduced in an abrupt and forced manner. The introduction fails to clearly articulate the problem being addressed, and the research questions presented feel contrived rather than naturally derived from real-world phenomena. The text is filled with self-coined concepts and terms that appear out of nowhere, making it difficult to follow the author’s underlying reasoning. Furthermore, the summary of contributions at the end of the introduction is obscure and convoluted, obscuring the key points. Additionally, the paper lacks a distinct, clear "Problem Statement" section to define the task and core concepts; instead, it conflates the definition of notation with the data construction process, neither of which is explained well, making the content extremely difficult to parse.

  2. The key intervention rules used in the benchmark are poorly defined, casting doubt on the persuasiveness of the substitution experiments. While the idea of ​​testing whether a model relies on spatiotemporal information by shifting event timestamps or swapping locations is sound, the paper fails to specify the exact magnitude of the temporal shifts or the criteria used to select alternative locations within the same city. Moreover, since the model never encounters samples with manipulated spatiotemporal data during training, the performance drop in "EventResidual" when tested on misaligned events may simply be due to the input deviating from the training distribution, and it does not necessarily prove that the model effectively utilizes spatiotemporal binding information.

  3. The experimental evaluation is too thin to support the claims made for a general-purpose benchmark. The core event evaluation only compares the authors' own model against a single baseline that was modified for this task. The paper provides no details on how this baseline was adapted, nor does it include comparisons with prominent predictive models from related literature or simple heuristic-based event prediction rules.

Questions for authors
  1. It is recommended to restructure the introduction and include a distinct Problem Statement section. Begin with intuitive, real-world examples to lead into the research question through a natural line of reasoning. Then, provide formal definitions for the core concepts and tasks. Additionally, revise the summary of contributions at the end of the introduction to be more direct and explicit, while minimizing the use of unnecessary, self-coined terminology.

  2. What are the specific rules for the temporal and spatial substitution tests? Have you considered the possibility that the performance degradation of EventResidual might stem from the substituted samples falling outside the training distribution? How do you plan to rule out this confounding factor?

  3. Could it include mainstream history-based forecasting models, large model-based approaches, and simple event-driven heuristic rules (as mentioned in the related work) in your evaluation? This would provide a more comprehensive comparison of how different types of methods perform on this benchmark.

2 comments
AuthorsOpenReview ↗

Response to Mrt2: problem definition, event controls, and evidence

We have reorganized the presentation around one task: ranking the next recorded POI from a user's observed check-in history, with external event information available at the prediction cutoff. The benchmark then asks both whether that information improves ranking and whether its specific time and location matter.

Task and construction. The introduction now starts with a concrete weather-warning example: a warning issued at 14:00 for conditions expected at 17:00–21:00 can inform a prediction made at 15:00. The issue time establishes when the information is available; the affected period and area describe where and when it is relevant. Section 3.1 separately defines the next-POI task, the training-known candidate vocabulary, and the prediction cutoff. Section 3.2 describes the dataset construction: Foursquare provides check-in histories and next-visit targets, while independently collected sports, weather, entertainment, and civic records provide event attributes. The next visited POI is not used to choose an event.

Specific time and location controls. An original event is selected using its information-availability evidence, the reference city, and overlap with the 24-hour event-retrieval window. Once selected, T translates the occurrence interval by +7 days in UTC, preserving duration and UTC weekday/hour. S changes the location to the recorded geometry of another real venue in the same city. The event type, time (for S), and consistently identity-masked description remain fixed, and affected derived features are updated. Neither change triggers a new event selection. Each pair preserves the same history, cutoff, target, candidate set, and fitted model (§3.3; Appendix B.1).

Testing the role of training exposure. In a separate study, we train models on either original events (G) or condition-matched replacements (R), then evaluate each model using both. For models trained on R, restoring G improves MRR@10 by +0.012143 for time (1,277 instances; adjusted 99.1667% CI [0.010319, 0.014682]) and +0.009677 for location (2,117 instances; [0.007947, 0.011667]). These are averages over three runs with six-comparison correction. They show that the original event information retains predictive value after exposure to this replacement mechanism. The main +7-day comparison remains a distinct evaluation (§5.4; Table 5; Fig. 5).

Evaluation across methods. We now report EventResidual, SASRec + Event, and task adaptations of AWARE and ELLMob. The latter two are described as adaptations because their candidate generation and next-POI interfaces differ from their source tasks. Appendix B.2 details the input, scoring, shortlist and fallback mechanisms; Tables 3–4 give paired counts and effects. EventResidual also has user- and episode-clustered intervals for its main contrasts. Together, the revised task definition and experiments make the purpose and scope of each event comparison explicit.

Thanks to the authors for the response and the updated paper. The rewritten introduction and the new problem definition section make the paper much easier to read, and the new results help with some of my concerns on the evaluation setup and baselines. I still think the experimental evidence as a whole is not strong enough for acceptance, but given the clear effort and improvements, I am raising my presentation score and overall rating from 1 to 2.

Reviewer_KBANOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BEYOND ROUTINE, a benchmark designed to evaluate whether next point-of-interest (Next-POI) prediction models correctly utilize the spatiotemporal information of external events. The authors aim to distinguish the predictive utility of event information from a model's dependence on correct spatiotemporal event bindings. To this end, they introduce four evaluation conditions: no event information (H), original event information (G), modified event times (T), and modified event locations (S). The benchmark combines user mobility records with independently collected event data. It requires event information to be available before the prediction cutoff and conducts paired comparisons while keeping the prediction instances, candidate locations, ground-truth labels, and trained models fixed. In addition, the authors propose a prediction model, EventResidual, and compare it with a task-adapted version of ELLMob. The experiments examine the predictive utility of event information for each model, their sensitivity to spatiotemporal event bindings, and their performance across different evaluation subsets.

Critical strengths
  1. The authors aim to distinguish the contribution of event information to prediction performance from a model's dependence on correct event times and locations. They introduce four evaluation conditions (H/G/T/S) to examine model sensitivity to spatiotemporal event information, extending the conventional comparison between conditions with and without event information. This is the paper's main contribution.

  2. The authors require event information to be available before the prediction cutoff and keep user histories, candidate locations, ground-truth labels, and trained models fixed in paired comparisons. This design helps reduce the influence of other factors on the results and improves comparability across different event-input conditions.

Critical weaknesses
  1. The primary contribution of this paper is an evaluation benchmark, yet the event-aware experiments evaluate only EventResidual and a task-adapted version of ELLMob. Although the paper reports different behaviors of the two models in the evaluation of spatiotemporal event bindings, evaluating only two systems is insufficient to establish the benchmark's applicability to different types of event-aware prediction models. This limits the evidence supporting the benchmark's broader diagnostic value.

  2. Section 3.3 and Appendix B.1 describe the general principles behind temporal and spatial substitutions, but the specific substitution rules and the criteria for constructing the stricter evaluation subsets remain insufficiently clear. In Section 5.4, the reported effect for EventResidual is +0.0354 under the standard temporal control condition, compared with −0.0067 on the Strict Temporal subset. Under the standard spatial control condition, the reported effect is +0.0305, whereas it is only +0.0022 on the Hard Spatial subset. Since these results are obtained from different evaluation instances, clearly specifying the selection criteria and construction rules for these subsets is important for interpreting the differences in results. Furthermore, the paper reports clustered confidence intervals for some of ELLMob's event-utility results but does not provide corresponding uncertainty estimates for the main results of EventResidual. In particular, for the Hard Spatial subset, the reported effect is only +0.0022. The current evidence is insufficient to determine the statistical reliability of this small positive effect.

Questions for authors
  1. Could the authors clarify the rationale for selecting EventResidual and ELLMob as representative evaluation targets? Are evaluation results available for other event-aware Next-POI prediction models that could further support the benchmark's applicability across different models?

  2. Could the authors explicitly specify the selection criteria and substitution rules for QTstrictQ_T^{strict} and QShardQ_S^{hard}, and clarify how these subsets differ from the standard QTQ_T and QSQ_S?

  3. Could the authors provide confidence intervals that account for clustering for the main paired-comparison results of EventResidual, particularly for the reported effect of +0.0022 on the Hard Spatial subset?

2 comments
AuthorsOpenReview ↗

Response to KBAN: evaluation across predictors and statistical reporting

The revised evaluation now includes four predictors with different ways of incorporating events. We have also made the paired evaluation populations and statistical evidence explicit, so that the comparisons can be read on their intended terms.

Broader method coverage. EventResidual combines history scores with event-dependent adjustments. SASRec + Event adds event features to a self-attentive sequence model. The AWARE adaptation combines a full-city behavioral prior with an LLM event residual, and the ELLMob adaptation turns event-conditioned generation into one-step POI ranking. Their paired changes in MRR@10 are:

PredictorG−HG−TG−S
EventResidual+0.000156 (18,055)+0.000347 (5,761)+0.000707 (11,094)
SASRec + Event+0.005334 (18,055)+0.004800 (5,761)+0.005056 (11,094)
AWARE adapted+0.010252 (4,958)+0.000270 (711)+0.000037 (4,957)
ELLMob adapted−0.009513 (4,529)−0.005987 (1,466)+0.006641 (2,775)

These results show why the three comparisons are useful together. ELLMob, for example, scores lower with the original event in the event-on/off and temporal comparisons, yet benefits from the original location in its spatial comparison. AWARE shows a different pattern: a positive event-on/off change with small time and location differences (Tables 3–4).

Evaluation interfaces and uncertainty. The first three predictors retain a full-city ranking. AWARE applies an LLM residual to at most 100 prior-ranked candidates, while ELLMob uses a fixed shortlist of up to 256 POIs; missed shortlist targets score zero, and fallback outputs remain in evaluation. AWARE and ELLMob are evaluated on the intersection of available paired predictions, with the number of instances shown in each table. Thus, each row is a within-method comparison, rather than a common-candidate leaderboard (Appendix B.2/C.1).

The updated EventResidual G−H, G−T, and G−S estimates now have user- and episode-clustered intervals. The user-clustered 95% intervals are [−0.000555, 0.000859], [−0.000890, 0.001642], and [−0.000459, 0.001850]. We report the other methods' paired changes as point estimates. The separate crossed training study also finds positive original-input gains after replacement-input training: +0.012143 for time and +0.009677 for location, with multiplicity-adjusted intervals (Fig. 5; Appendix C.3).

Evaluation sets. The revision centers on the three primary populations: event on/off (18,055 instances), time (5,761), and location (11,094). We have removed the former strict-temporal and hard-spatial auxiliary analyses; those labels denoted additional sample filters, not stronger changes to an identical set of instances. Their earlier estimates are therefore not used as evidence for the revised main results.

Thank you for your feedback, I have adjusted the score.

Reviewer_g6UDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BEYOND ROUTINE, a controlled benchmark for evaluating event utility and temporal/spatial binding dependence in next-POI prediction. It uses independently documented events and fixed-instance comparisons with event removal and time/place substitutions. The authors also propose EventResidual, which shows positive paired effects on the primary evaluation sets, although binding effects vary under stricter controls.

Critical strengths
  • The fixed-instance H/G/T/S comparisons provide a useful distinction between event utility and temporal and spatial binding dependence, beyond context-on/off evaluation.
  • Prediction-time availability evidence and target-independent event assignment address important evaluation concerns. The benchmark also reveals differences across models and evaluation populations.
Critical weaknesses
  • The validity of the binding comparisons needs further justification. Event assignment uses city bindings and a 24-hour retrieval window (Equation 2), while T and S substitute event time and place information. It remains unclear whether the resulting effects reflect dependence on supported bindings or sensitivity to the substituted inputs.
  • Binding effects vary substantially across evaluation populations. The strict temporal effect is negative (−0.0067), while the hard spatial effect is small (+0.0022). The paper appears to report clustered intervals for ELLMob, but I could not find corresponding intervals for EventResidual.
Questions for authors
  • How can the authors distinguish the effects of incorrect event time/place bindings from those of implausible event inputs in the G/T/S comparisons?
  • Could the authors clarify whether clustered intervals are available for EventResidual and explain the negative strict temporal effect and small hard spatial effect?
  • Can the G/H/T/S protocol be evaluated with additional event-aware next-POI methods, such as AWARE, to demonstrate its broader applicability?
3 comments
AuthorsOpenReview ↗

Response to g6UD: event comparisons, training exposure, and uncertainty

The central question in your review is whether the time and location comparisons reveal how a predictor uses an event, rather than simply how it reacts to altered inputs. We have clarified the construction of these comparisons and added a crossed training study that directly examines the role of exposure to replacement inputs.

Event selection and comparisons. We first select an event using information available at the prediction cutoff, a city match, and overlap with a 24-hour event-retrieval window. The next visited POI is never used for this selection. On the selected event, T moves the occurrence interval forward by seven days in UTC; S substitutes the recorded geometry of another real venue in the same city. The remaining event content is kept, identifying names are masked consistently, and derived features are recomputed. Each pair keeps the user's history, cutoff, target, candidate set, and fitted predictor fixed (§3.3; Appendix B.1).

Training with replacement inputs. The crossed study trains separate models on original event inputs (G) and condition-matched replacements (R), then evaluates each model with both inputs. Of particular interest is RG − RR: the gain from providing G to a model trained on R.

AttributeInstancesRG − RR (MRR@10)Adjusted 99.1667% CI
Time1,277+0.012143[0.010319, 0.014682]
Location2,117+0.009677[0.007947, 0.011667]

The results average three training runs; the intervals account for user–episode dependence and six comparisons. Original event time and location retain predictive value after the model has encountered the corresponding replacement mechanism during training. This is a separate, condition-matched analysis, while the main temporal evaluation uses the fixed seven-day shift (§5.4; Table 5; Fig. 5).

Uncertainty and wider applicability. For the updated EventResidual run, the paired G−H, G−T, and G−S effects are +0.000156, +0.000347, and +0.000707 MRR@10. Their user-clustered 95% intervals are [−0.000555, 0.000859], [−0.000890, 0.001642], and [−0.000459, 0.001850]; episode-clustered intervals are also reported. The revised paper interprets these small main-study effects in light of their uncertainty. We also evaluate SASRec + Event and an AWARE adaptation alongside EventResidual and ELLMob, with paired sample sizes and candidate interfaces stated explicitly (Tables 3–4; Appendix B.2/C.1).

AuthorsOpenReview ↗

Follow-up on Event Time and Location Comparisons

We would like to briefly follow up on your concern about whether the G/T/S comparisons reflect the use of event time and location, rather than a response to unfamiliar inputs.

In the revised manuscript (§5.4, Table 5, and Figure 5), we added a crossed training–evaluation experiment. Models are trained separately with original and condition-matched replacement inputs, then evaluated under both conditions. Importantly, even after training with replacement inputs, restoring the original event information improves MRR@10 by +0.012143 for time and +0.009677 for location. The corresponding adjusted 99.1667% confidence intervals are [0.010319, 0.014682] and [0.007947, 0.011667].

These results provide additional evidence that the original event time and location retain predictive value after exposure to the replacement mechanism. The crossed experiment is distinct from the main seven-day temporal comparison. We have also clarified the time/location replacement rules and added user- and episode-clustered confidence intervals for EventResidual.

We hope these additions help address your concern about the interpretation of the event comparisons. We would be glad to clarify any remaining questions about the revised evaluation.

Thank you for your additional experiments and the revised manuscript. I think the rebuttal addresses most of my concerns. However, I noticed the EventResidual result have changed in Table 3. I think it is not small change and I couldn't find the reason. Therefore, I will keep my rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revision summary: clearer benchmark protocol and expanded evaluation

We have uploaded the revised manuscript and replied to the three reviews. The revision clarifies how external events are linked to next-POI prediction and extends the evaluation of their time and location information.

The main changes are:

  1. A clearer benchmark protocol. Section 3 separates the task definition, independent event collection, information availability, and fixed-instance time/location comparisons.
  2. Evaluation across four predictors. Tables 3–4 report EventResidual, SASRec + Event, AWARE adaptation, and ELLMob adaptation, with paired sample counts and implementation details in Appendix B/C.
  3. A crossed training study. After training with replacement inputs, restoring original event time and location improves MRR@10 by +0.012143 and +0.009677, respectively. Both have positive multiplicity-adjusted 99.1667% intervals (§5.4; Table 5; Fig. 5).

The main result tables and conclusions have been updated to reflect the revised estimates and clustered uncertainty. The reviewer-specific replies provide the relevant numbers and explanations, with section references for further detail.

AuthorsOpenReview ↗

Final Rebuttal Summary and Appreciation

Dear Area Chair and Reviewers,

We sincerely thank you for your time, careful evaluation, and constructive feedback throughout the review and discussion process. Your comments have been valuable in helping us identify weaknesses in the original submission and improve the paper.

During the rebuttal period, we made several substantive additions and revisions to address the main concerns raised by the reviewers.

First, we conducted a new crossed training–evaluation study to examine whether the observed effects of temporal and spatial substitutions could be explained by unfamiliar inputs. We also expanded our evaluation to four predictors by adding SASRec + Event and an adaptation of AWARE, providing a broader examination of the benchmark across different modeling approaches. In addition, we supplemented the experimental results with clustered confidence intervals and clearer reporting of evaluation populations.

Beyond these experiments, we substantially revised the manuscript to clarify the problem formulation, event selection and substitution procedures, baseline adaptations, and interpretation of the results. We also refined our conclusions to better reflect the available statistical evidence and the limitations of the evaluation.

We believe these changes have meaningfully improved the clarity, experimental coverage, and rigor of the paper. More importantly, the discussion has helped us develop a more careful understanding of what our benchmark can demonstrate and where its limitations remain.

We are grateful to all reviewers for their thoughtful questions, suggestions, and engagement during the rebuttal. We also thank the Area Chair for overseeing the review process and considering our work.

Thank you again for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.