- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces STEPPY, a new model for forecasting wildlife movement by leveraging movement patterns across spatiotemporal scales and species. Using a transformer, STEPPY encodes key forecasting information: the forecast horizon, taxonomic information, population-level spatial scale, and previous trajectory observations with day, time, and location. These observations are represented with relative spatiotemporal encoding and then combined with random Fourier basis features to predict a spatial probability distribution of where the animal will be. STEPPY outperforms several baselines on the MoveBench dataset across all setups, showing that cross-study training conditioned on long historical contexts strongly benefits wildlife movement modeling.
Critical strengths
- The paper is well written, motivated, and clear, making it enjoyable to read
- Proposing novel approaches to improve and scale wildlife movement forecasting is crucial to guide conservation policies better.
- The temporal flexibility and good performance of STEPPY across setups, with clear improvements of each of its components shown through a well-executed ablation study.
Critical weaknesses
Generally, my main concern is that the paper lacks a bit of ambition, in my opinion:
- The paper builds on top of MoveBench, an unpublished paper uploaded to arXiv 10 days before the ICLR deadline, following a similar structure and reusing some of its content, for example, the performance plot figure, which is the same in the two papers, except for the additional STEPPY curve (Figure 2 in STEPPY and Figure 3 in MoveBench). This makes the paper seem incremental compared with the dataset paper. I therefore expect an innovative methodology with in-depth analysis and/or very strong performance to distinguish this paper from the dataset paper. I concede here that the gains are quite important ~20%, but on the other hand, the authors also admit in the abstract that the state-of-the-art baselines “only modestly outperform simple statistical approaches”, which may question this statement of L419: “high predictive accuracy can be achieved without explicit access to environmental information”. In particular, the energy score is somewhat difficult to interpret, and there are only a few (potentially cherry-picked) qualitative examples in both the main text and Appendix. Adding more examples and/or experiments could help convince me that the performance gains are significant and translate into different conservation decisions.
- To follow up on the previous point, the reason I think the performance could even be higher is the choice not to use environmental variables. I agree that STEPPY outperforming the baselines without explicit environmental information is remarkable, but I see a missed opportunity to improve performance further by integrating this valuable information, which is a key driver of wildlife movements. Additionally, wildlife movement modeling is also about trying to understand the relationships between movement and the environmental drivers. Relying only on past trajectories seems conceptually insufficient to model wildlife movement well. I believe Figure 4b suggests that combining MoveFormer's ability to incorporate environmental variables for more fine-grained predictions with STEPPY's ability to better leverage historical context would improve prediction. I therefore see no strong reason not to include this information, especially since it is readily available in MoveBench, and the transformer could be trained to be robust to missing environmental variables when they are unavailable or too heavy to use, as has been done recently in ecological modeling.
Questions for authors
I would like to hear the author's opinion on the concerns mentioned above, whether they consider them valid, and whether they will conduct additional experiments to address them. I believe the paper is valuable, and I am strongly willing to increase my rating if the authors address my concerns.
2 comments
I appreciate the effort made by the authors to address my concerns, particularly the additional experiments in Appendices D and E, which demonstrate the relevance of Steppy to ecology and conservation. As the authors rightly point out, "Steppy’s largest improvements over other methods are most pronounced in cases where an animal is resident (relatively stationary in a local region for several successive movements), returning to a previously-visited location, or maintaining its direction of travel with respect to recent movements." This also suggests that Steppy could potentially benefit from incorporating environmental variables, particularly in situations where past movement patterns are less informative, or trajectories are less predictable. I am therefore somewhat disappointed that environmental variables have not been considered. To me, this represents a step backward compared to previous approaches, rather than merely a direction for future work, as suggested by the authors. I also remain unconvinced by the computational argument, as I believe there are relatively straightforward ways to address these challenges while also ensuring robustness to missing environmental data. Incorporating such variables would, in my view, make Steppy a more reliable tool for practitioners.
Despite these reservations, I still believe the paper makes a valuable contribution. My assessment would ideally fall between 2 and 3, but given the limited granularity of this year's rating scale, I have decided to round up and increase my rating from 2 to 3.
Thank you very much for your thoughtful review and feedback. Part of W1 (Interpretability of energy score) is related to a request for a breakdown of the metric requested by Reviewer sb62, and W2 (Using environmental information) was also a concern raised by Reviewer xfdy. We have addressed both in the shared response above. Please feel free to raise any additional questions or points for clarification in either thread. To address your additional concerns:
W1 (Dataset vs. method): MoveBench is a prior work that proposes a benchmark for animal movement forecasting. Our work evaluates on that benchmark and explicitly references it as prior work. Steppy is a distinct and novel methodological contribution, it is the first method to enable multi-scale multi-species movement forecasting, and, as you pointed out, it achieves very strong empirical results when compared to the baselines proposed in MoveBench. We agree that calling this “high predictive accuracy” is imprecise and have updated the language to “higher predictive accuracy than previously demonstrated,” but we do stand by the observation that prior deep learning methods for animal movement forecasting “only modestly outperform simple statistical approaches,” as this does seem to be true empirically. As discussed in Section 4.1, the next-best method at every timescale (i.e. the best possible existing method other than Steppy, selected per-timescale) is only 6% better than a correlated random walk on the MoveBench temporal test set and 2% better than a correlated random walk on the MoveBench individual test set. We think that this speaks to the difficulty of the probabilistic wildlife forecasting task. Steppy proposes a new kind of approach that is empirically a big step forward and we hope can help unlock even more progress for the field.
W1 (Experiments and qualitative examples): Thank you for this suggestion. We have added new quantitative results breaking down prediction cases with potential relevance to ecology and conservation (Appendix Section D.2), as well as a significant number of new qualitative results (Appendix Section E). For the prediction breakdown, we have focused on five common movement scenarios that can be identified by heuristics applied to ground truth trajectories: revisitation, site arrival and departure, continuation, and residence (see Section D.2 for precise definitions). Breaking down test cases by these scenarios has provided additional insights, in particular that Steppy’s largest improvements over other methods are most pronounced in cases where an animal is resident (relatively stationary in a local region for several successive movements), returning to a previously-visited location, or maintaining its direction of travel with respect to recent movements. These complement our other results that indicate the utility of Steppy’s contextual processing, which takes recent observations into account at prediction time more effectively than prior work.
Then, we have added qualitative comparisons of every method, both on the ecologically-relevant scenarios described above (in Section E.2), as well as on randomly-selected examples across all of MoveBench (in Section E.1). For the latter, we randomly selected examples from both the temporal and individual test sets, grouped by the same taxonomic and range groupings in Figure 2, resulting in 24 pages of qualitative prediction comparisons. We have found these examples very helpful for building intuition about the behavior of the different methods on various forecasting cases, and thank you for your suggestion.