- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 1
Read review
Paper summary
This paper investigates the impact of urban events on the prediction of the next POI, focusing on whether models---while leveraging event information to improve prediction performance---actually utilize the specific temporal and spatial details of those events. To this end, the authors construct a benchmark called BEYOND ROUTINE, which links check-in trajectories with independently collected event records. Using this benchmark, they evaluate the utility of event information and the extent to which models rely on temporal and spatial data by systematically removing events, altering their timing, or replacing their locations within the prediction samples. Furthermore, the authors propose a reference model, EventResidual, which combines scoring based on historical trajectories with a gating mechanism for event-based adjustments, and conduct comparative experiments against modified baseline models using this benchmark.
Critical strengths
The Related Work section is very clearly structured. It provides a well-organized overview, ranging from datasets on human mobility to the classification of methods for next-location prediction, and the inclusion of a comparative table helps readers quickly grasp the progress made in this field.
Critical weaknesses
-
The introduction and the problem definition section are not reader-friendly, with concepts introduced in an abrupt and forced manner. The introduction fails to clearly articulate the problem being addressed, and the research questions presented feel contrived rather than naturally derived from real-world phenomena. The text is filled with self-coined concepts and terms that appear out of nowhere, making it difficult to follow the author’s underlying reasoning. Furthermore, the summary of contributions at the end of the introduction is obscure and convoluted, obscuring the key points. Additionally, the paper lacks a distinct, clear "Problem Statement" section to define the task and core concepts; instead, it conflates the definition of notation with the data construction process, neither of which is explained well, making the content extremely difficult to parse.
-
The key intervention rules used in the benchmark are poorly defined, casting doubt on the persuasiveness of the substitution experiments. While the idea of testing whether a model relies on spatiotemporal information by shifting event timestamps or swapping locations is sound, the paper fails to specify the exact magnitude of the temporal shifts or the criteria used to select alternative locations within the same city. Moreover, since the model never encounters samples with manipulated spatiotemporal data during training, the performance drop in "EventResidual" when tested on misaligned events may simply be due to the input deviating from the training distribution, and it does not necessarily prove that the model effectively utilizes spatiotemporal binding information.
-
The experimental evaluation is too thin to support the claims made for a general-purpose benchmark. The core event evaluation only compares the authors' own model against a single baseline that was modified for this task. The paper provides no details on how this baseline was adapted, nor does it include comparisons with prominent predictive models from related literature or simple heuristic-based event prediction rules.
Questions for authors
-
It is recommended to restructure the introduction and include a distinct Problem Statement section. Begin with intuitive, real-world examples to lead into the research question through a natural line of reasoning. Then, provide formal definitions for the core concepts and tasks. Additionally, revise the summary of contributions at the end of the introduction to be more direct and explicit, while minimizing the use of unnecessary, self-coined terminology.
-
What are the specific rules for the temporal and spatial substitution tests? Have you considered the possibility that the performance degradation of EventResidual might stem from the substituted samples falling outside the training distribution? How do you plan to rule out this confounding factor?
-
Could it include mainstream history-based forecasting models, large model-based approaches, and simple event-driven heuristic rules (as mentioned in the related work) in your evaluation? This would provide a more comprehensive comparison of how different types of methods perform on this benchmark.
2 comments
Thanks to the authors for the response and the updated paper. The rewritten introduction and the new problem definition section make the paper much easier to read, and the new results help with some of my concerns on the evaluation setup and baselines. I still think the experimental evidence as a whole is not strong enough for acceptance, but given the clear effort and improvements, I am raising my presentation score and overall rating from 1 to 2.
Response to Mrt2: problem definition, event controls, and evidence
We have reorganized the presentation around one task: ranking the next recorded POI from a user's observed check-in history, with external event information available at the prediction cutoff. The benchmark then asks both whether that information improves ranking and whether its specific time and location matter.
Task and construction. The introduction now starts with a concrete weather-warning example: a warning issued at 14:00 for conditions expected at 17:00–21:00 can inform a prediction made at 15:00. The issue time establishes when the information is available; the affected period and area describe where and when it is relevant. Section 3.1 separately defines the next-POI task, the training-known candidate vocabulary, and the prediction cutoff. Section 3.2 describes the dataset construction: Foursquare provides check-in histories and next-visit targets, while independently collected sports, weather, entertainment, and civic records provide event attributes. The next visited POI is not used to choose an event.
Specific time and location controls. An original event is selected using its information-availability evidence, the reference city, and overlap with the 24-hour event-retrieval window. Once selected, T translates the occurrence interval by +7 days in UTC, preserving duration and UTC weekday/hour. S changes the location to the recorded geometry of another real venue in the same city. The event type, time (for S), and consistently identity-masked description remain fixed, and affected derived features are updated. Neither change triggers a new event selection. Each pair preserves the same history, cutoff, target, candidate set, and fitted model (§3.3; Appendix B.1).
Testing the role of training exposure. In a separate study, we train models on either original events (G) or condition-matched replacements (R), then evaluate each model using both. For models trained on R, restoring G improves MRR@10 by +0.012143 for time (1,277 instances; adjusted 99.1667% CI [0.010319, 0.014682]) and +0.009677 for location (2,117 instances; [0.007947, 0.011667]). These are averages over three runs with six-comparison correction. They show that the original event information retains predictive value after exposure to this replacement mechanism. The main +7-day comparison remains a distinct evaluation (§5.4; Table 5; Fig. 5).
Evaluation across methods. We now report EventResidual, SASRec + Event, and task adaptations of AWARE and ELLMob. The latter two are described as adaptations because their candidate generation and next-POI interfaces differ from their source tasks. Appendix B.2 details the input, scoring, shortlist and fallback mechanisms; Tables 3–4 give paired counts and effects. EventResidual also has user- and episode-clustered intervals for its main contrasts. Together, the revised task definition and experiments make the purpose and scope of each event comparison explicit.