- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces TimeArrowMob, a controlled benchmark for studying temporal directionality in human mobility prediction. It modifies transition counts through circulation constraints and generates trajectories using Eulerian circuits. This construction provides exact direction labels and computable reference strengths while matching selected statistics across directions and strength levels. The paper evaluates nine existing methods using prediction accuracy, directional response R, and historical-order benefit G.
Critical strengths
The main contribution is a controlled way to vary directional information while keeping several potentially confounding statistics unchanged, including location frequencies, undirected transition counts, and self-transition counts. The combination of circulation constraints and Eulerian circuits makes these controls exact. This provides a useful basis for examining temporal directionality beyond ordinary prediction accuracy.
Critical weaknesses
The evaluated methods do not show consistently positive R and G across strength levels. However, the experiments do not include a model known to use the injected directional information, so it remains unclear whether these measurements can reliably detect such use.
Questions for authors
Can the authors add an oracle given reference mechanism and a history-only transition-count baseline to demonstrate that the visible histories contain usable directional information and that the proposed metrics detect its use?
Assessing Directional Evidence and Benchmark Sensitivity
Yes. Your suggestion identifies a gap in our evaluation: the last-location baseline shows what zero response looks like, but it does not establish whether the 45-location histories contain usable directional evidence or whether R detects a predictor acting on that evidence. We have implemented the two controls you suggest.
Reference-informed control. The predictor receives the two reference transition matrices P+ and P−, but neither the source-direction label nor the next location. For each history, it sums log(Pij+/Pij−) over the 44 observed transitions. We first evaluate how well this score distinguishes windows drawn from forward versus reverse trajectories, using within-user ROC AUC at each strength level. The predictor then uses the transition matrix selected by the score to predict the most likely next location. This separates two questions: can the history reveal its source orientation, and does using that information change the top-1 prediction? Even a distinguishable orientation need not change the prediction if both matrices favor the same next location.
History-only control. This predictor receives no reference matrices or direction labels. It predicts the most frequent successor of the current location among transitions observed in the visible history. If the history contains no earlier departure from that location, it predicts the current location again.
We will evaluate both controls on the original paired queries, using the same R, the same eligible histories and interventions for G, and user-level confidence intervals. The full-cohort estimates are not yet available. Until we have them, our conclusion is limited to what we observed in the nine frozen predictors; their results alone do not establish the sensitivity of the benchmark.