- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1(−1)
Read review
Paper summary
This work addresses the kinematic similarity between mobility domains like pedestrians, vehicles, vessels and aircraft. This work proposes a predictor that exploit the kinematic similarity and achieves zero-shot transfer performance in each mobility domain.
Critical strengths
This work focuses on an interesting problem, which is zero-shot transfering to different mobility domains, by using the kinematic simlarity. The designed framework targets to adapt to the characteristic of different mobility domain and achieves with a relatively light weight model.
Critical weaknesses
-
Normalization for baseline methods. As the scale in different mobility domain is a huge difference, are baselines also normalized by domain-specific scale? It would be helpful to show the advantage of the proposed method, if the normalization is applied in the preprocessing step for all baselines. Otherwise, will feel the main performance drop can come from scale difference.
-
Table 4 provides the squared error for both zero-shot and in-domain. a) it would be helpful to provide same metrics for baseline methods, to show the effectiveness of the zero-shot transfer ability more clearly. b) Although the squared error could represent the prediction error, it is necessary to show the same for ADE/FDE value, for in-domain as well, as ADE/FDE is used as main evaluation metrics in this work.
-
About comparison between Table 1 and 3. In Table 3, ETH/UCY and SDD are put in the same fold for evaluation. Intuitively, the performance in Table 3 can be worse, as the most similar data source (ETH/UCY, or SDD) is removed from training data. As shown in Table 3, for the proposed method, removing SDD makes performance on ETH/UCY worse, but removing ETH/UCY makes performance on SDD same or even better in FDE. And similar patterns happen in some baselines for both datasets, which shows adding similar human trajectory data for training does not improve, or even downgrade the prediction accuracy. It would be helpful to provide the analysis of the results.
-
The motivation is not clear enough. In introduction, it says: a new sensing platform or agent class lacks a predictor, and collecting enough data takes time. There is no concrete example provides, and in the experiments, all the evaluated mobility domains like aircraft or vehicle already have good amounts of data. Also even the method is targeting sparse data domain, the proposed method still requires observed data for scale computing.
-
Following in point 4, it would be helpful to include evaluation of data-efficiency, that in-domain models are trained with an increasing number of trajecotries in this domain, and compared with the zero-shot results. This could show that, up to how many trajectories provided in target domain, the proposed method under zero-shot transfer outperforms in-domain training.
Questions for authors
Same as in weaknesses.
6 comments
W4 & W5. We rewrote the motivation in the introduction around the cold-start period of a new domain, and we added two experiments that measure what this period requires and how long it lasts. When a port radar begins to track a new vessel class, or a drone fleet starts in its first city, a few tracks are available on the first day, but the hundreds of trajectories that training needs take weeks or months to collect. KineTraj is meant for this period.
In such a new domain, the three numbers that KineTraj needs, the characteristic scales of step length, speed and acceleration, can be computed from observed past windows. The table gives the change of the zero-shot minADE₂₀ when these scales are computed from n randomly drawn observed windows in place of all test windows, averaged over ten draws and three model seeds. A dash means that a draw contained only stationary windows, for which the step scale is zero.
| Observed windows n | ETH/UCY | SDD | CT-DMA | T-Drive | GeoLife | Aircraft |
|---|---|---|---|---|---|---|
| 1 | +11.1% | +14.1% | +15.9% | +34.0% | – | – |
| 2 | +6.0% | +6.6% | +7.6% | +18.5% | +30.3% | +1.2% |
| 5 | +0.7% | −0.6% | +2.1% | +9.2% | +11.0% | +0.8% |
| 10 | +1.3% | +1.1% | +1.0% | +8.8% | +11.3% | +0.7% |
| 20 | −0.3% | +0.1% | +0.6% | +1.6% | +1.3% | +0.6% |
| 50 | −0.2% | −0.3% | +0.3% | +0.8% | +2.3% | +0.2% |
| 100 | −0.4% | −0.1% | +0.2% | +0.3% | +1.0% | +0.2% |
With only 20 randomly drawn windows in place of all test windows, the zero-shot error changes by less than 2% on all six folds.
In addition, we added a data-efficiency experiment. For each held-out domain, KineTraj is trained in domain on the first N training trajectories of that domain, with N from 10 to the full split, and compared with the zero-shot model, which sees no trajectory of the domain. The results are given below as minADE₂₀ averaged over three seeds. The last row is the N at which in-domain training matches the zero-shot model, found by interpolation of the curve.
| In-domain trajectories N | ETH/UCY | SDD | CT-DMA | T-Drive | GeoLife | Aircraft |
|---|---|---|---|---|---|---|
| 10 | 0.457 | 0.585 | 1.780 | 560 | 26.2 | 1185 |
| 30 | 0.421 | 0.562 | 1.479 | 506 | 24.1 | 1097 |
| 100 | 0.274 | 0.472 | 1.166 | 396 | 19.6 | 935 |
| 300 | 0.241 | 0.361 | 1.021 | 344 | 17.5 | 821 |
| 1000 | 0.233 | 0.346 | 0.869 | 329 | 16.7 | 730 |
| all | 0.228 | 0.332 | 0.797 | 308 | 15.0 | 713 |
| Zero-shot KineTraj (N = 0) | 0.261 | 0.349 | 1.223 | 368 | 17.2 | 813 |
| N at which in-domain matches zero-shot | ≈160 | ≈800 | ≈80 | ≈180 | ≈470 | ≈330 |
With 10 labelled trajectories of the target domain, the in-domain model is 46–75% worse than the zero-shot one, and with 30 it is still 21–61% worse. In-domain training catches up only at about 80 to 800 trajectories, depending on the domain (last row of the table). The full curve is in Appendix B.4 (Table 12 and Figure 5) of the revision.
We hope this addresses all concerns and thank them again for the suggestions, which have strengthened the paper.
Thanks the authors for the detailed response. The additional results resolves W2, W3 and W5 well. For W1, the concern of the normalization still remains. The baselines are not evaluated with and without normalization. LSTM, Transformer, CVAE and GAN are reported to use the same normalization scale as the proposed method, but when on the in-range folds like ETH/UCY or SDD, their ADE is about 2 times that of constant velocity model, while on a very difference scale source, like Aircraft scenario, it jumps to about 10 times, which shows the normalization may not effectively remove the scale difference for these baselines. For W4, the practicality and real-world use case of the proposed method are still not convincingly motivated. For example, a new vessel class still belongs to the existing vessel domain, where existing vessel data would be the better and natural choice rather than cross-domain transfer from pedestrian data. I will keep my rating.
We thank the reviewer for the careful and specific comments. We address the five weaknesses in order below.
W1. All baselines are normalized by domain scale. We thank the reviewer for pointing this out. Appendix A.4 of the revision now describes how each baseline is normalized. EqMotion and SingularTrajectory operate on coordinates divided by a per-domain median step length. UniTraj uses a per-domain z-score of displacements. OmniTraj is run with its scale-normalization option. Chronos and Time-MoE standardize each window. The LSTM, Transformer, CVAE and GAN baselines use the same per-domain coordinate normalization as KineTraj. The gap in Table 1 therefore does not come from a difference in scale.
W2. We added an experiment that trains KineTraj and EqMotion in domain and compares each with its zero-shot version in ADE and FDE. An in-domain model is trained on the training split of the held-out dataset itself, under the same cap of 2,000 trajectories as each source domain. The results are given below as minADE₂₀ / minFDE₂₀, in metres and in nautical miles for CT-DMA. The zero-shot rows repeat Table 1 and, like the in-domain KineTraj row, are means of three seeds.
Zero-shot KineTraj is within 5–20% of its in-domain version in ADE on ETH/UCY, SDD, T-Drive, GeoLife and Aircraft, and 53% behind it on CT-DMA, the vessel fold. Without any training data from the target domain, it has a lower ADE and FDE than EqMotion trained on that domain on ETH/UCY, SDD and Aircraft, and EqMotion trained in domain is better on CT-DMA, T-Drive and GeoLife. With the same in-domain training data, KineTraj has a lower ADE and FDE than EqMotion on all six folds, by 1–17% in ADE. These results are in Appendix B.2.2 (Table 9) of the revision.
W3. Similar pedestrian data does help, and how much it helps depends on whether a method can also use the data of other families. Removing the other pedestrian set raises the error in 13 of the 16 comparisons between Table 1 and Table 4 of the revision. The four coordinate-only models (LSTM, Transformer, CVAE and GAN) rise by 13–96%. They transfer poorly across families, so the other pedestrian set is the only source they can use. KineTraj, EqMotion and SingularTrajectory are the three trained methods that transfer best in Table 1, and they change by only −2% to +5%, because the vessel, vehicle and aircraft data already give them what the other pedestrian set gives.
We checked this with a new nearest-neighbour analysis. For each evaluation window of a pedestrian set, we take the 50 most similar source training windows in KineTraj's normalized input space. The table gives the share of each source dataset, in percent, among these 50 windows, among the 50 nearest windows in raw coordinates, and among all source training windows.
In the normalized space, 82–87% of the most similar windows come from the vessel, vehicle and aircraft sets. In raw coordinates the other pedestrian set would supply 65–69% of them, although it holds under 5% of the source windows.
The three cases that do not get worse are within about two seed standard deviations. For KineTraj on SDD the change is 0.000 in ADE and −0.011 in FDE, against seed standard deviations of 0.010 and 0.029. SingularTrajectory on ETH/UCY changes by −0.008 in ADE against 0.013, and UniTraj on SDD by −0.144 against 0.074. The results therefore give no evidence that similar data lowers the accuracy.
We have added this analysis to Appendix B.1 of the revision.