acceptodds
Under review as a conference paper at ICLR 2027

Think on Visual Trajectories: Transferable Trajectory Recovery via Drifting

Abstract

Recovering dense trajectories from sparse observations supports urban mobility analysis, commercial decision-making, and regional planning. However, reliance on city-specific road-segment vocabularies limits the transferability of many existing methods. Inspired by how human experts reconstruct journeys on maps, we investigate visual maps as a shared language for cross-city trajectory recovery: observed locations are rendered on a map, and missing movement is recovered through video generation. We construct a dataset pairing these map-rendered sparse observations with videos of the corresponding dense trajectories. Building on this formulation, we propose DriftTraj, a lightweight framework that adapts Drifting Models to map-conditioned trajectory recovery. It produces each video block in a single forward pass, avoiding iterative diffusion sampling, and recovers longer trajectories through successive blocks. Because trajectory markers occupy only a small fraction of each frame, we introduce a differentiable red-marker extractor and incorporate its spatial features into a geometric training objective, providing explicit supervision for marker location and spatial structure. We evaluate DriftTraj on Porto, Beijing, Xi'an, and Chengdu, training the generator only on Porto and transferring it to the other three cities with zero-shot transfer and limited-data fine-tuning. The results support visual maps as a shared representation for cross-city trajectory recovery and demonstrate the potential of one-step visual generation for transferable mobility modeling.

Primary area:
learning on time series and dynamical systems
Keywords:
Trajectory Recovery, Generative Model, Drift

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tfFyOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DriftTraj, a framework that reformulates trajectory recovery as map-conditioned video generation. Sparse observations are rendered as markers on OpenStreetMap tiles, and missing movement is recovered by generating video frames in which a red marker traces the inferred path. The generator is trained via Drifting Models, enabling one-forward-pass generation per video block without iterative diffusion sampling. A differentiable red-marker extractor provides explicit geometric supervision for marker localization, and a Truth Forcing mechanism re-renders markers on the original map between autoregressive steps to prevent appearance drift. The model is pretrained exclusively on Porto and transferred zero-shot to Beijing, Xi'an, and Chengdu, with optional limited-data fine-tuning. Experiments across four cities and nine baselines support visual maps as a shared cross-city representation for trajectory recovery.

Critical strengths
  • Novel and well-motivated problem formulation. The central insight — that visual maps provide a city-agnostic shared representation, bypassing the road-segment vocabulary bottleneck of graph-based methods — is both original and compelling. The analogy to how human experts trace paths on maps is intuitive and well-articulated. This reformulation opens a genuinely new design space for trajectory recovery and related mobility tasks.

  • The differentiable marker-aware geometric supervision is a technically elegant solution to a real problem. The observation that RGB reconstruction loss alone may reproduce the static map background without accurately localizing the small moving marker is well-identified. The red-response extractor with its position-dependent centroid gradient provides a principled, differentiable path from pixel-level generation to explicit location supervision, avoiding the need for discrete marker detection that would break gradient flow. The ablation confirming that removing this loss causes early error spikes and instability validates its necessity.

  • The efficiency argument is well-supported and practically significant. The 198.8x speedup over a video diffusion reference, with 10,000 trajectories processed in 11.07 minutes, demonstrates that one-step generation makes vision-based trajectory recovery practicable rather than merely theoretical. This is an important feasibility result for the broader visual trajectory research program.

  • Comprehensive experimental design with honest framing. The four-city evaluation with zero-shot transfer and limited-data fine-tuning covers the key transfer scenarios. The paper is commendably transparent that the Top@5 (Oracle) results are diagnostic rather than deployable, and that Recall is lower due to the visual generation paradigm. The appendices provide substantial additional analysis — alternative pretraining city, direct visual regression comparison, matched-data-budget comparison, seed stability — that strengthens the empirical foundation beyond the main table.

Critical weaknesses
  • The coordinate recovery pipeline is insufficiently documented in the main text. The paper's central claim is about recovering trajectories, but the critical step of converting generated video frames back to GPS coordinates is barely mentioned in the body — Truth Forcing only states that the marker position is extracted. The actual geographic calibration (pixel-to-GPS transform via Web Mercator fitting) is relegated to the appendix with minimal detail. Given that this step determines the final MAE/RMSE metrics, its accuracy, failure modes, and sensitivity to map rendering parameters should be discussed in the main paper. A reader who does not consult the appendix cannot fully assess the method's validity.

  • Road-segment Recall is consistently and substantially lower than strong baselines. On Porto, DriftTraj's Mean@5 Recall (66.59%) trails RNTrajRec (67.29%) and MM-STGED (67.52%). On Chengdu, it falls below RNTrajRec by over 2 percentage points (82.02% vs. 84.17%). Even after fine-tuning, Beijing Recall (54.46%) remains below several baselines. The paper attributes this to generating trajectories over the full map image rather than performing road-segment classification, but this explanation is offered without analysis of when or why the visual approach fails to capture road-level structure. For applications where road-level accuracy matters (e.g., traffic flow estimation, turn-by-turn routing), this is a material limitation that is insufficiently addressed.

  • Beijing performance is dramatically weaker than other cities, and the cause is not analyzed. Under zero-shot transfer, Beijing's MAE (281.98m) is roughly 2.7x higher than Xi'an (94.98m) and 2.4x higher than Chengdu (115.72m). Even after fine-tuning, Beijing MAE (272.19m) remains far above the other two target cities. The paper notes that Beijing's nominal sampling interval is 60 seconds (versus 12–15 seconds for the other cities), but does not systematically analyze whether this alone explains the gap or whether Beijing's urban structure (larger road grid, different topology) poses specific challenges to the visual representation. Understanding this failure mode is important for assessing the limits of cross-city transferability.

  • The cross-city transfer claim assumes matched rendering settings, which is a strong but unvalidated constraint. The paper acknowledges in the appendix that results assume a consistent rendering protocol — including map style, zoom-level settings, and marker colors — and that invariance to different basemap providers, zoom levels, or marker colors remains unevaluated. This is a significant scope limitation: the "shared visual representation" claim is only demonstrated under a narrow rendering regime. If zoom level or map style changes between cities — which is plausible in real deployment — the transferability may not hold. At minimum, a sensitivity analysis over zoom levels or map styles would substantially strengthen the claim.

  • Unmatched model capacity and resolution. The generator has only 8M parameters operating at 96x96 pixel resolution — both are quite low. While the paper acknowledges that limited compute prevented systematic investigation of larger models and higher resolutions, the absence of any scaling analysis makes it impossible to assess whether the method's limitations (particularly Recall and Beijing performance) are fundamental to the visual approach or artifacts of insufficient capacity. A single experiment at 192x192 or with a moderately larger model would provide valuable signal.

Questions for authors
  • Beijing gap analysis. Is the large Beijing MAE primarily explained by the 60-second sampling interval (creating longer gaps between observations), or do Beijing-specific urban structure and map rendering also contribute? Have you tried subsampling Porto trajectories to a 60-second interval to isolate the effect of observation sparsity from city-specific factors?

  • Recall improvement path. Given that road-segment Recall is consistently below graph-based baselines, is there a hybrid approach that could combine the transferability advantages of visual generation with road-network constraints for segment-level accuracy? For instance, could map-matching be applied more aggressively, or could the generator be conditioned on road geometry features within the visual input?

  • Sensitivity to rendering parameters. How sensitive is transfer performance to zoom level? If the Porto-trained model is evaluated on Beijing map tiles rendered at a different zoom level than training, how much does MAE degrade? This would directly test the robustness of the "shared visual representation" claim.

  • Observation density effects. All experiments use a fixed retention ratio of 0.1. How does performance scale with observation density — for example, at 0.05 (extremely sparse) or 0.3 (moderately dense)? Does the relative advantage of the visual approach over graph-based baselines change with sparsity?

  • Candidate selection without ground truth. The Top@5 (Oracle) results show large improvements over Mean@5, but require ground truth for selection. For real deployment, is there a practical selection mechanism — for example, using trajectory likelihood under a frozen model, or consistency with observed road structure — that could approximate oracle performance without ground truth?

  • Non-vehicular trajectories. The evaluation uses exclusively taxi datasets. Has the method been tested on pedestrian or cycling trajectories, which have different movement patterns (shorter segments, more frequent direction changes, less road-network constraint)? The visual map representation might behave very differently for these modalities.

7 comments
AuthorsOpenReview ↗

Response to Reviewer tfFy (1/5)

We thank Reviewer tfFy for the careful review. We respond to each point below. All changes in the revised PDF are marked in blue.

Response to Weakness 1:

We agree that this step directly affects the final metrics and should be explained in the main text. We have added it to the main text of the revision (end of Section 4.3), with the full formulas in Appendix A.5. The recovery pipeline consists of, in order: marker decoding, temporal alignment, and a deterministic pixel-to-geographic transform.

First, the marker's pixel position is extracted from each generated frame using the red response and weighted centroid defined in the paper. If extraction fails, the previous position is kept. Temporal resampling and coordinate alignment are then performed under the observation constraints, yielding a predicted pixel position for each query time. The geographic transform is calibrated from the pixel/GPS pairs of the known observations. The fit does not use the ground-truth coordinates of missing positions.

Let SS be the side length of the original square image and r=96r=96 the side length of the network input. A known pixel point (ui,vi)(u_i,v_i) is first mapped to the network pixel scale with the vertical axis flipped:

xi=(rSui,−rSvi).\mathbf x_i=\left(\frac{r}{S}u_i,-\frac{r}{S}v_i\right).

The corresponding known longitude and latitude (λi,φi)(\lambda_i,\varphi_i) are projected to Web Mercator, with angles in radians and R=6,378,137R=6{,}378{,}137 m:

\left( R\lambda_i, R\log\tan\left(\frac{\pi}{4}+\frac{\varphi_i}{2}\right) \right).$$ Because the maps are north-up square crops, we fit only a shared scale $s$ and a 2D translation $\mathbf b$, rather than an arbitrary affine transform: $$(s,\mathbf b)=\arg\min_{s > 0, \mathbf b} \sum_i\left\Vert\mathbf m_i-(s\mathbf x_i+\mathbf b)\right\Vert_2^2.$$ The closed-form solution is $$s= \frac{\sum_i(\mathbf x_i-\bar{\mathbf x})^\top(\mathbf m_i-\bar{\mathbf m})} {\sum_i\Vert\mathbf x_i-\bar{\mathbf x}\Vert_2^2}, \qquad \mathbf b=\bar{\mathbf m}-s\bar{\mathbf x}.$$ For a predicted pixel $(\hat u_t,\hat v_t)$ at the network scale, we first compute $$(\hat X_t,\hat Y_t)=s(\hat u_t,-\hat v_t)+\mathbf b,$$ and then recover longitude and latitude through the inverse projection: $$\hat\lambda_t=\frac{\hat X_t}{R}, \qquad \hat\varphi_t=2\arctan\left(\exp\left(\frac{\hat Y_t}{R}\right)\right)-\frac{\pi}{2}.$$ Finally, radians are converted to degrees.
AuthorsOpenReview ↗

Response to Reviewer tfFy (2/5)

Response to Weakness 3, Question 1, and Question 4:

We added statistics on the per-pixel geographic scale:

CityMean scale (m/pixel)Median (m/pixel)Min–Max (m/pixel)
Porto24.9220.064.04–112.27
Xi'an29.4224.284.60–97.40
Beijing29.1721.735.10–200.41
Chengdu28.5921.806.07–103.02

The mean scales of the four cities are close, so Beijing's error cannot simply be attributed to a larger meters-per-pixel scale. We therefore tested different observation retention ratios γ\gamma on the four cities. The table below reports Mean@5 results. Each cell is Recall / MAE / RMSE, and the γ=0.1\gamma=0.1 column is taken from the original paper.

Cityγ=0.05\gamma=0.05γ=0.1\gamma=0.1 (original paper)γ=0.3\gamma=0.3
Porto60.19 / 153.41 / 194.3266.59 / 110.41 / 146.5479.22 / 47.99 / 72.79
Xi'an78.86 / 151.29 / 196.9585.84 / 94.98 / 130.5792.36 / 34.97 / 56.00
Beijing50.97 / 319.93 / 410.1352.35 / 281.98 / 372.5763.93 / 158.48 / 234.25
Chengdu74.43 / 172.82 / 221.5682.02 / 115.72 / 157.2491.96 / 41.61 / 65.92

As observations become sparser, coordinate errors rise and Recall falls on all four cities, which shows that observation sparsity is an important factor in recovery quality. As you noted, the nominal sampling interval of the raw Beijing trajectories is 60 s. At γ=0.1\gamma=0.1, the gap between observations is about 600 s, and where a vehicle goes within 10 minutes is highly uncertain. Porto's nominal interval is 15 s, so reaching the same observation gap would require γ≈0.025\gamma\approx0.025, at which point a large drop in performance is to be expected. We therefore believe that the sparser observations of the Beijing data are an important cause of this gap.

To directly separate the effect of observation sparsity from that of the city itself, we followed your suggestion and downsampled the Porto trajectories from the original 15 s to 60 s (matching Beijing's nominal interval), then re-evaluated with γ=0.1\gamma=0.1 unchanged. Each cell below is Recall / MAE / RMSE.

SettingMean@5Top@5
Original 15 s, scored against 15 s ground truth (original paper)66.59 / 110.41 / 146.5475.54 / 89.77 / 120.02
Downsampled to 60 s, scored against 60 s ground truth47.27 / 167.34 / 224.8760.72 / 124.97 / 171.73
Downsampled to 60 s, scored against original 15 s ground truth55.07 / 198.29 / 244.4969.48 / 152.07 / 192.25

The second row is the most comparable to the Beijing setting. After downsampling to 60 s, Porto's Mean@5 MAE rises from 110.41 m to 167.34 m and its Recall falls from 66.59% to 47.27%, even below Beijing's 52.35%. This shows that the sampling interval is one of the main factors behind the Beijing gap.

Regarding Question 4, on whether the advantage over baselines changes with observation density, we added Top@5 results under different retention ratios (each cell is Recall / MAE / RMSE, and the γ=0.1\gamma=0.1 column is taken from the original paper):

Cityγ=0.05\gamma=0.05γ=0.1\gamma=0.1 (original paper)γ=0.3\gamma=0.3
Porto71.44 / 121.43 / 156.6675.54 / 89.77 / 120.0284.66 / 41.73 / 62.92
Xi'an87.36 / 123.49 / 162.1291.64 / 80.72 / 110.6296.03 / 30.07 / 47.23
Beijing63.35 / 249.77 / 325.5765.01 / 218.20 / 294.0174.41 / 121.88 / 183.91
Chengdu84.95 / 141.11 / 182.4889.46 / 94.47 / 131.6595.75 / 35.61 / 55.87

We also re-evaluated the strongest baseline, RNTrajRec, on Porto under the same retention ratios:

Method (Porto)γ=0.05\gamma=0.05γ=0.3\gamma=0.3
RNTrajRec62.0 / 141.8 / 185.179.0 / 45.3 / 71.9
DriftTraj Mean@560.19 / 153.41 / 194.3279.22 / 47.99 / 72.79
DriftTraj Top@5 (Oracle)71.44 / 121.43 / 156.6684.66 / 41.73 / 62.92

At γ=0.3\gamma=0.3, DriftTraj Mean@5 is on par with RNTrajRec (slightly higher Recall, slightly higher MAE and RMSE). At γ=0.05\gamma=0.05, Mean@5 falls behind RNTrajRec. When observations are very sparse, the five candidates tend to follow different routes, and pointwise averaging pulls the result away from all of them. Top@5 outperforms RNTrajRec on all three metrics in both settings, and its margin is larger at γ=0.05\gamma=0.05 (20.4 m lower MAE, 9.4 percentage points higher Recall) than at γ=0.3\gamma=0.3 (3.6 m lower MAE). This indicates that the sparser the observations, the more valuable the diversity of the generated candidates becomes, and the more is lost by simple averaging. Better use of the candidates without ground truth, such as the road fusion described below, is therefore most useful in the sparse regime. These results have been added to the revision (Appendix J, Tables 16–18).

AuthorsOpenReview ↗

Response to Reviewer tfFy (3/5)

Response to Weakness 2, Question 2, and Question 5:

We agree that the lower road-level Recall limits the method's use in segment-level applications. The generator can currently learn the road layout from the visual input, but the training objective does not explicitly require predicted points to respect the connectivity of the directed road network. In addition, pointwise averaging over multiple candidates can pull the output to somewhere between two candidate roads. We believe these are the main reasons for the lower Recall, and the revision now states them in Section 5.2.

To explore combining visual generation with road constraints, and to address your Question 5 on how to make use of multiple candidates without ground truth, we added a road-fusion method that requires no ground truth for the missing positions. It keeps the five candidates generated by the frozen model and builds an additional reference route from the known anchors and an external directed road network.

Specifically, longitude and latitude are first projected onto each city's local UTM metric plane. Candidate road positions are identified near adjacent known anchors, and a connecting route is searched on the directed road network. The cost of road edge ee is

\qquad \rho=0.5,\quad v_0=50,$$ where $L_e$ is the edge length and $v_e$ is a fixed priority value assigned by road class. It expresses a preference over road classes and is not a measured travel speed. The search also takes into account the distance from each anchor to its candidate road positions. (We use $\rho$ for the road-weight exponent to avoid confusion with the observation retention ratio $\gamma$.) Once the route is determined, we interpolate along its arc length according to the relative time between adjacent observations, obtaining $\mathbf p_{\mathrm{road}}(t)$. The five model trajectories are projected onto the same metric plane and averaged to give $\bar{\mathbf p}_{\mathrm{model}}(t)$, and we then compute $$\mathbf p_{\mathrm{fused}}(t) =w\mathbf p_{\mathrm{road}}(t) +(1-w)\bar{\mathbf p}_{\mathrm{model}}(t), \qquad w=0.75.$$ Finally, the known observed positions are restored and the result is projected back to longitude and latitude. $\rho$ and $w$ are selected on the Porto validation set and are not tuned again when transferring to other cities, so all cities share the same values. If the road search cannot connect two anchors, we fall back to a direct connection. The road-fusion results are shown below. Each cell is Recall / MAE / RMSE. "Best baseline" takes, for each metric, the best value among all baselines in Table 1 of the revision, and the last column is the relative gain of the fused result over it (increase in Recall, reduction in MAE and RMSE). | City | Best baseline | DriftTraj Mean@5 | DriftTraj + road fusion | Improvement over best baseline (%) | |---|---:|---:|---:|---:| | Porto | 69.84 / 102.6 / 140.6 | 66.59 / 110.41 / 146.54 | 74.81 / 102.11 / 137.59 | 7.1 / 0.5 / 2.1 | | Xi'an | 85.91 / 93.1 / 126.5 | 85.84 / 94.98 / 130.57 | 90.96 / 75.11 / 103.45 | 5.9 / 19.3 / 18.2 | | Beijing | 56.61 / 278.6 / 373.7 | 52.35 / 281.98 / 372.57 | 60.99 / 230.77 / 306.78 | 7.7 / 17.2 / 17.9 | | Chengdu | 84.17 / 95.1 / 131.8 | 82.02 / 115.72 / 157.24 | 90.59 / 87.49 / 119.45 | 7.6 / 8.0 / 9.4 | After fusion, Recall, MAE, and RMSE improve over Mean@5 on all four cities, and the fused result is better than the best baseline value on every metric in every city. Relative to the best baseline, Recall increases by 5.9–7.7%, MAE decreases by 0.5–19.3%, and RMSE decreases by 2.1–18.2%. This shows that in applications where a road network is available, a road prior can further improve the output of the visual model, and the whole procedure requires no ground truth. It is reported in the revision as an additional hybrid variant (Appendix K, Table 19), clearly separated from the purely visual method.
Reviewer_Rmp9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

DriftTraj formulates sparse trajectory recovery as map-conditioned video generation to support transfer across cities without city-specific road vocabularies. A compact generator trained with a drifting objective predicts four frames per call. Recovery combines autoregressive generation, map redrawing, temporal alignment, endpoint correction, and pointwise averaging of five candidates. Experiments evaluate recovery across cities, zero-shot transfer, inference cost, and component ablations, with additional controls and target-city sample-budget comparisons in the appendix.

Critical strengths

The shared visual representation provides a clear approach to cross-city transfer (I think it's new). Map redrawing and marker-aware geometric supervision address concrete difficulties in making the video-based pipeline usable for trajectory recovery.

The overall performance seems good, e.g., the Beijing zero-shot result is encouraging: MAE is 281.98 m versus 318.2 m for the best baseline in the main table.

Critical weaknesses
  1. The practically advantageous operating regime remains insufficiently established. The comparison includes learned road-constrained methods, but lacks a conventional pipeline combining HMM-style route inference from sparse observations with interpolation along the recovered route. This is particularly relevant to cross-city deployment without learning a road vocabulary. Zero-shot MAE improves in Beijing but does not beat linear interpolation in Xi'an or Chengdu. The reported five-candidate runtime is 11.07 minutes per 10,000 trajectories, versus approximately two minutes for non-visual baselines; the 198.8x speedup concerns Wan. Moreover, the 96x96 maps and 20 training timestamps, followed by interpolation back to the original timestamps, leave the scope for larger geographic extents, complex routes, and long gaps unclear.

  2. The information and geometric constraints supplied by generation versus postprocessing are not sufficiently disentangled. Under the single-frame conditioning described in Section 4.2 and Appendix A, repeated visits to B in A→B→C→B→D may present the same current image despite different next anchors. The outer procedure knows the next observation, so this is a conditioning clarification, not evidence that the complete system fails. Recall is measured after HMM map matching, while the reported Truth Forcing ablations do not isolate endpoint correction. These results therefore do not directly establish the feasibility of the final coordinates before map matching or the contribution of generated route structure.

Questions for authors
  1. Evidence on applicability: What evidence identifies a regime where the quality, target-city data requirements, and end-to-end cost justify this approach relative to conventional road-based recovery? Please relate the current results to geographic extent, route complexity, and observation-gap length; the concern is the advantageous operating regime, not dominance everywhere. In addition, considering the cost, would you please give idea about the trade-off between cost and benefit? I think using a video-based model for this task may not be so worthing compared with simple map matching-like tradicitonal methods.

  2. Evidence on postprocessing: How large are endpoint errors and correction-induced displacements, and how does road deviation or motion continuity change after correction and Mean@5 averaging? Under q₀=aⱼ and the stated linear interpolation, selecting ℓ*=1 makes the corrected local segment exactly linear interpolation between observations. How frequently does this occur? Its prevalence and isolated correction diagnostics could materially change the interpretation of what the generator contributes.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Rmp9 (1/3)

We thank Reviewer Rmp9 for the careful review. We respond to each point below. All changes in the revised PDF are marked in blue.

Response to Question 1 (applicable regime) and Weakness 1 (transferability and time cost):

Based on the additional experiments above, the regime where we believe it is worth using is: the target city has no trajectory data available for training, or the observation gaps are medium or long. We explain this point by point.

  1. In a new city without training data, learned road-based methods (e.g., RNTrajRec, MM-STGED) need dense trajectories and a road network in the target city for training, and must be retrained for every new city. DriftTraj needs only the OSM tiles of the target city, and the model trained on Porto can be used as is. In this setting, the real alternatives are Linear and HMM, which also require no training. Relative to HMM + route interpolation, DriftTraj Mean@5 reduces MAE by 17.6% on Porto (134.04 → 110.41), by 33.8% on Xi'an (143.56 → 94.98), and by 24.3% on Chengdu (152.81 → 115.72), and is on par on Beijing. Xi'an, Chengdu, and Beijing are all zero-shot. Compared with Linear, the advantage depends on the gap length, which also gives a simple usage rule. DriftTraj outperforms Linear in the medium- and long-duration groups. In the long-duration group, MAE is reduced by 21.1% on Porto, by 18.6% on Beijing, and by about 6% on Xi'an and Chengdu.

  2. Visual generation and the road network are not an either-or choice, they are complementary. For settings where a road network is available, we fused the DriftTraj output with a road-network route in a simple way (no ground truth required, the parameters are selected on the Porto validation set and then kept fixed, details are in our response to Reviewer tfFy). Each cell below is Recall / MAE / RMSE. "Best baseline" takes, for each metric, the best value among all baselines in Table 1 of the revision (including HMM), and the last column is the relative gain of the fused result over it (increase in Recall, reduction in MAE and RMSE).

CityBest baselineDriftTraj Mean@5DriftTraj + road fusionImprovement over best baseline (%)
Porto69.84 / 102.6 / 140.666.59 / 110.41 / 146.5474.81 / 102.11 / 137.597.1 / 0.5 / 2.1
Xi'an85.91 / 93.1 / 126.585.84 / 94.98 / 130.5790.96 / 75.11 / 103.455.9 / 19.3 / 18.2
Beijing56.61 / 278.6 / 373.752.35 / 281.98 / 372.5760.99 / 230.77 / 306.787.7 / 17.2 / 17.9
Chengdu84.17 / 95.1 / 131.882.02 / 115.72 / 157.2490.59 / 87.49 / 119.457.6 / 8.0 / 9.4

The fused result is better than the best baseline value on every metric in every city. Relative to the best baseline, Recall increases by 5.9–7.7%, MAE decreases by 0.5–19.3%, and RMSE decreases by 2.1–18.2%. In other words, even when a road network is already available, the generated trajectory still provides useful information on top of it. We believe this is the most direct answer to whether the approach is worthwhile compared with conventional map-matching-style methods.

  1. Learned baselines are indeed faster at inference (about 2 minutes), but they require hours or even days of training, plus the corresponding training data, in every new city, whereas the extra cost of DriftTraj is a one-off few minutes of inference. In real trajectory-tracing scenarios, a single recovered route is often not enough. Practitioners frequently want several feasible solutions to compare and judge manually. A generative model provides exactly these candidates, which is why we report Top@5.

The supporting results have been added to the revision: the HMM baseline in Table 1, the scale and duration analysis in Appendix I, and the road fusion in Appendix K. We also hope this work can serve as a starting point for image-based trajectory recovery and offer some inspiration for follow-up research.

AuthorsOpenReview ↗

Response to Reviewer Rmp9 (2/3)

Response to Weakness 1 (geographic extent and complex routes):

We added statistics on the geographic extent and duration covered by the current evaluation. On a 96×9696\times96 grid, the scale multiplied by the image side length is roughly the geographic span of one side of the map, so the test sets contain map regions of varying sizes.

CityMean scale (m/pixel)Median scale (m/pixel)Scale min–max (m/pixel)Mean duration (s)Median duration (s)Duration min–max (s)
Porto24.9220.064.04–112.27412.7337560–1560
Xi'an29.4224.284.60–97.40505.6742060–2292
Beijing29.1721.735.10–200.41430.9236060–2400
Chengdu28.5921.806.07–103.02415.2536060–1752

We further grouped trajectories by meters-per-pixel scale and by total duration, and compared DriftTraj with Linear (each cell is MAE / RMSE). DriftTraj outperforms Linear in the medium- and high-scale groups and in the medium- and long-duration groups, while Linear is better in the low-scale and short-duration groups. The reason is intuitive: over short distances, a straight-line connection is already fairly accurate, and the extra computation of generative recovery may not be worth it. When motion must be recovered over a larger area and a longer time span, the advantage of the visual representation shows. These results have been added to the revision (Appendix I, Tables 14 and 15).

CityScale groupRange (m/pixel)LinearDriftTraj Mean@5
PortoHigh27.55–112.27184.21 / 246.69145.51 / 194.20
PortoMedium15.12–27.47136.78 / 182.64108.56 / 144.34
PortoLow4.04–15.1261.64 / 80.7477.05 / 100.93
Xi'anHigh33.44–97.40127.50 / 177.05122.07 / 171.14
Xi'anMedium17.84–33.4497.23 / 126.6890.57 / 123.69
Xi'anLow4.60–17.7657.77 / 77.4272.22 / 96.78
BeijingHigh31.01–200.41561.93 / 731.00456.93 / 591.22
BeijingMedium15.85–30.86298.06 / 401.63243.75 / 325.38
BeijingLow5.10–15.80115.80 / 160.37144.75 / 200.46
ChengduHigh30.78–103.02162.81 / 218.09154.80 / 215.29
ChengduMedium16.08–30.78114.98 / 152.69110.94 / 149.27
ChengduLow6.07–16.0840.64 / 53.4981.29 / 106.97

The results grouped by total duration are as follows. This also explains your observation that the overall zero-shot MAE does not beat Linear on Xi'an and Chengdu: in both cities, DriftTraj outperforms Linear in the medium- and high-scale groups and in the medium- and long-duration groups, and the overall gap comes mainly from the short-distance, short-duration groups.

CityDuration groupRange (s)Mean duration (s)LinearDriftTraj Mean@5
PortoLong480–1560675.04174.22 / 236.70137.46 / 183.51
PortoMedium285–480375.00140.25 / 184.19108.44 / 144.48
PortoShort60–285187.3468.19 / 89.2185.24 / 111.52
Xi'anLong564–2292865.54124.17 / 170.39116.23 / 164.06
Xi'anMedium324–564434.0599.96 / 132.8995.66 / 130.20
Xi'anShort60–324216.3258.38 / 77.8972.98 / 97.36
BeijingLong420–2400748.38551.94 / 711.02449.53 / 575.13
BeijingMedium240–420346.49311.92 / 419.17255.98 / 338.39
BeijingShort60–240196.94111.96 / 162.87139.95 / 203.59
ChengduLong468–1752718.71152.82 / 204.77143.24 / 200.44
ChengduMedium264–468357.08122.24 / 162.63117.02 / 157.32
ChengduShort60–264169.0543.40 / 56.9186.81 / 113.82
AuthorsOpenReview ↗

Response to Reviewer Rmp9 (3/3)

Response to Weakness 1 (HMM baseline):

Following your suggestion, we added a conventional recovery pipeline that requires no trajectory training in the target city. It builds an HMM from the distance between each observation and its candidate roads and the coherence between adjacent matches, decodes the matched sequence with Viterbi, connects the matched positions with shortest routes, and interpolates along the route in proportion to time.

The table below compares this HMM baseline with the DriftTraj results from the original paper. Each cell is Recall / MAE / RMSE. On Porto, Xi'an, and Chengdu, DriftTraj Mean@5 outperforms HMM on all three metrics. The Beijing results are close: HMM has a slightly lower MAE, while DriftTraj Mean@5 has a slightly lower RMSE. The fine-tuned DriftTraj is better on all three metrics. These results have been added to the revision (Table 1, with the implementation described in Appendix B).

CityHMM + route interpolationDriftTraj Mean@5DriftTraj (FT) Mean@5
Porto65.84 / 134.04 / 181.6866.59 / 110.41 / 146.54—
Xi'an77.52 / 143.56 / 203.4385.84 / 94.98 / 130.5787.34 / 87.96 / 120.75
Beijing53.87 / 278.61 / 373.7152.35 / 281.98 / 372.5754.46 / 272.19 / 359.37
Chengdu74.40 / 152.81 / 213.4282.02 / 115.72 / 157.2484.42 / 104.44 / 141.50

Response to Weakness 2 and Question 2:

We ablated the two coordinate-correction stages separately: segment-level endpoint correction, and anchor alignment after restoring the full timeline. Going from both stages off to both on, MAE drops by 9.26 m, RMSE drops by 7.48 m, and Recall rises by 3.87 percentage points, so both stages contribute. Meanwhile, the raw coordinates before any correction (both stages off) differ from the final results by less than 10 m, which indicates that the directly generated coordinates are already reasonably reliable. This ablation has been added to the main text of the revision (Section 5.3, Table 2).

We would also like to clarify the role of map matching in our metrics. DriftTraj outputs free-space coordinates, and MAE and RMSE are computed directly on these coordinates, with no map matching involved. HMM map matching is used only for Recall, because Recall is defined over road segments and a free-space output must first be mapped to segments before it can be computed. The MAE and RMSE in the table therefore already reflect the quality of the final coordinates before map matching. This is now stated in Section 5.1 of the revision.

SettingMean@5 MAE ↓Mean@5 RMSE ↓Recall ↑
Both corrections off119.67154.0262.72
Segment endpoint correction only112.74147.5865.39
Full-timeline alignment only111.88148.2466.44
Both on (original paper)110.41146.5466.59

Regarding ℓ⋆=1\ell^\star=1, we agree with your mathematical conclusion: in this case the linear interpolation operator makes the deviation term exactly zero, and the corrected local segment does reduce to linear interpolation. We measured how often this happens. One-step truncation occurs mostly when the observation gap is very short. Across the four cities, there is no trajectory with missing points for which all missing points in all 5 candidates reduce to straight lines at the same time. Linear degeneration therefore covers only a small fraction of missing points, and the vast majority of predicted positions are still determined by the shape of the generated path.

CitySegments with ℓ⋆=1\ell^\star=1Affected missing points in a single candidateMissing points affected in all 5 candidates
Porto2,356/16,015 (14.71%)2.46%1.07%
Xi'an6,423/23,355 (27.50%)8.46%3.49%
Beijing61/6,030 (1.01%)0.44%0%
Chengdu4,404/19,580 (22.49%)6.27%2.58%

Regarding repeated visits to the same location, you are right that this ambiguity exists under single-frame conditioning: when the vehicle passes near B twice, the current image seen by the generator can be nearly identical. At present this ambiguity is mitigated by the outer procedure: the next known anchor and the observation times are used by the outer recovery procedure for segment-wise generation, local endpoint selection, and the subsequent anchor alignment, so the complete pipeline still makes use of these observation constraints. We list explicit temporal and target-anchor conditioning inside the generator as a direction for future improvement.

Thank you again for your detailed comments.

Authors of Paper 16248

Reviewer_C1S2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the challenge of cross-city transferability in trajectory recovery without relying on city-specific road-segment vocabularies. To tackle this problem, the authors propose DriftTraj, which introduces a novel paradigm inspired by how human experts reconstruct journeys on maps. Specifically, experts typically reconstruct trajectories by locating observed positions on a map and tracing plausible paths between them. Although road layouts vary across cities, the visual language of maps remains recognizable. Following this intuition, DriftTraj generates a block of video frames in a single forward pass and recovers longer trajectories by generating successive blocks. A differentiable red-marker extractor is introduced to provide supervision on marker locations and spatial structures. Experiments on four datasets demonstrate the zero-shot cross-city transferability of DriftTraj and support the potential of visual maps as a shared representation for trajectory recovery.

Critical strengths
  1. The paper presents a novel perspective on improving the cross-city transferability of trajectory recovery models by reformulating trajectory recovery as a video generation task. It leverages the shared visual representation of road maps across different cities to reduce the influence of city-specific road layouts and improve generalization.

  2. The paper provides a comprehensive experimental evaluation, including comparisons with various baselines across multiple cities, zero-shot cross-city transfer experiments, and extensive ablation studies. The additional analyses of model components and parameter settings further demonstrate the effectiveness of the proposed method.

Critical weaknesses
  1. The necessity of formulating trajectory recovery as video generation is not sufficiently justified. The paper converts continuous GPS coordinates into map images, generates future RGB frames, and then extracts marker positions to recover the original coordinates. Although this formulation is novel, it introduces an indirect prediction pipeline for a task whose final output is only two-dimensional coordinates. The authors have compared DriftTraj with direct coordinate regression in Appendix G, but this comparison does not fully establish whether generating complete RGB frames is necessary. A more convincing evaluation would compare DriftTraj against a map-conditioned trajectory generator that directly predicts continuous coordinates while retaining the same visual information.
  2. The method may be sensitive to map resolution and geographic scale. DriftTraj represents trajectories using fixed 96×96 map images. However, the geographic distance represented by each pixel can vary depending on the map extent and zoom level. Consequently, identical physical movements may correspond to substantially different pixel displacements, while nearby roads may become indistinguishable at coarse spatial resolutions. Although the differentiable marker extractor supports subpixel localization, this does not eliminate information loss caused by rasterization. The authors should investigate how different rendering resolutions and geographic scales affect both localization accuracy and cross-city transfer performance.
Questions for authors
  1. How is the geographic extent of each 96×96 map image determined? Do maps from different cities represent comparable real-world distances per pixel, or does the geographic scale vary across trajectories? This clarification is important for understanding how the model handles differences in spatial scale during cross-city transfer.
  2. Have the authors evaluated the sensitivity of DriftTraj to map resolution or zoom level? In particular, does the model maintain its recovery accuracy when the same trajectory is rendered at different geographic scales?
4 comments
AuthorsOpenReview ↗

Response to Reviewer C1S2 (1/2)

We thank Reviewer C1S2 for the careful review. We respond to each point below. All changes in the revised PDF are marked in blue.

Response to Weakness 1:

Your understanding of our pipeline is correct. We use the RGB representation so that the road layout, the observed locations, and the motion to be recovered are all modeled in one spatial representation. This removes the need to rebuild a road-segment vocabulary for every target city.

Following your suggestion, we implemented a model that receives the same visual information and uses a similar architecture, but outputs coordinates directly. It takes the current map/observation frame and noise as input, and predicts the next 4 consecutive 2D coordinates in a single forward pass (normalized by image width and height and concatenated into an 8-dimensional vector). It is trained with the same batch-independent drifting loss, and full trajectories are obtained by repeated rollouts. The training time matches that of our model, and we likewise report Mean@5 and Top@5 over 5 outputs. Each cell is MAE / RMSE.

CityDirect-coordinate model Mean@5DriftTraj Mean@5Direct-coordinate model Top@5DriftTraj Top@5
Porto140.35 / 182.76110.41 / 146.54121.03 / 155.9689.77 / 120.02
Xi'an107.87 / 144.6594.98 / 130.5793.86 / 123.7180.72 / 110.62
Beijing305.14 / 398.15281.98 / 372.57239.64 / 313.70218.20 / 294.01
Chengdu127.45 / 168.80115.72 / 157.24109.12 / 142.6894.47 / 131.65

On all four cities, DriftTraj achieves lower MAE and RMSE under both aggregation schemes, which shows that generating full visual frames does help. One likely reason is that generating the map background together with the motion marker helps the model learn spatial structure. This model has been added to the revision as an additional baseline (Appendix G, Table 12).

Response to Weakness 2 and Question 1:

Each map is rasterized to a fixed 96×9696\times96 grid, but the geographic extent it covers varies with the trajectory, so the meters-per-pixel scale is not strictly the same across samples or cities. We apologize for not making this clear in the original paper. We have added the following statistics to the revision (Appendix A.1, Table 5).

CityMean scale (m/pixel)Median (m/pixel)Min–Max (m/pixel)
Porto24.9220.064.04–112.27
Xi'an29.4224.284.60–97.40
Beijing29.1721.735.10–200.41
Chengdu28.5921.806.07–103.02

The mean scales are close across cities, but the range within each city is wide. You are right that subpixel extraction improves localization on the given raster but cannot recover road details lost during rasterization. This is now stated in Appendix A.1 of the revision.

AuthorsOpenReview ↗

Response to Reviewer C1S2 (2/2)

Response to Question 2:

We first split each dataset into three equal-sized groups (low, medium, high) by each trajectory's meters-per-pixel scale. The results are shown below (each cell is MAE / RMSE). The metric error grows with the scale, i.e., the results degrade to some extent as the scale increases. These results have been added to the revision (Appendix I, Table 14).

CityScale groupRange (m/pixel)Mean@5Top@5
PortoHigh27.55–112.27145.51 / 194.20121.99 / 162.08
PortoMedium15.12–27.47108.56 / 144.3487.27 / 116.88
PortoLow4.04–15.1277.05 / 100.9359.96 / 80.97
Xi'anHigh33.44–97.40122.07 / 171.14107.39 / 149.16
Xi'anMedium17.84–33.4490.57 / 123.6977.53 / 105.49
Xi'anLow4.60–17.7672.22 / 96.7857.16 / 77.09
BeijingHigh31.01–200.41456.93 / 591.22366.45 / 482.42
BeijingMedium15.85–30.86243.75 / 325.38182.11 / 250.24
BeijingLow5.10–15.80144.75 / 200.46105.60 / 148.81
ChengduHigh30.78–103.02154.80 / 215.29131.13 / 183.39
ChengduMedium16.08–30.78110.94 / 149.2790.83 / 126.26
ChengduLow6.07–16.0881.29 / 106.9761.34 / 85.15

Beyond the tile zoom level z=14z=14 used in training, we also tested z=13z=13 and z=15z=15. The table below reports Top@5 results. Each cell is Recall / MAE / RMSE, and the z=14z=14 column is taken from the original paper. Compared with z=14z=14, MAE increases by 6.48–15.23 m and RMSE by 8.73–17.93 m. In other words, the model adapts to zoom-level changes to some degree, but it is not zoom-invariant.

Cityz=14z=14z=13z=13z=15z=15
Porto75.54 / 89.77 / 120.0271.99 / 102.51 / 135.0873.32 / 98.74 / 131.67
Xi'an91.64 / 80.72 / 110.6290.54 / 88.11 / 120.1790.30 / 87.20 / 119.35
Beijing65.01 / 218.20 / 294.0163.26 / 228.24 / 306.2162.88 / 227.13 / 304.36
Chengdu89.46 / 94.47 / 131.6586.79 / 105.19 / 142.4086.45 / 109.70 / 149.58

In practice, fixing the rendering protocol is a simple way to control input differences across cities: z=14z=14 tiles for anywhere in the world are readily available from OpenStreetMap. The revision now states that the rendering protocol should be kept fixed (Appendix A.1), and we will require the same in the code. Joint training over multiple scales and map styles is a direction for further improving robustness. These results have been added to the revision (Appendix A.1, Table 6).

Thank you again for your suggestions. These additional experiments gave us a clearer picture of the benefit of the visual representation, the effect of geographic scale, and where rendering robustness still needs to improve.

Authors of Paper 16248

Thanks for authors detailed rebuttals, which have addressed my main concerns. I have improved my score.

AuthorsOpenReview ↗

Response to Reviewer C1S2

Thank you for reading our rebuttal and for raising your score. We are glad that you see value in reformulating trajectory recovery as video generation, and in using the visual language of maps. Your questions directly improved the paper. The map-conditioned model that outputs coordinates directly gave a much cleaner test of whether generating full frames helps, and the meters-per-pixel statistics, scale groups, and zoom-level experiments now make the role of geographic scale explicit. Thank you again for the thoughtful review.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the first revision

We thank the three reviewers for their careful reading and for the concrete experiments they suggested. Carrying them out gave us a much clearer picture of where a visual approach to trajectory recovery helps and where it does not. In the revised PDF, every addition is shown in blue. Below we summarize the main changes, ordered by importance.

  1. We now state the value of the visual formulation (Reviewers Rmp9 and C1S2). Its target setting is a city with no trajectories available for training and medium or long observation gaps. In that setting the realistic alternatives are linear interpolation and HMM-based map matching, since learned road-based methods must be retrained for every city. Grouping the test sets by geographic scale and by duration (App. I, Tables 14 and 15) shows that DriftTraj beats linear interpolation in the medium and large groups of all four cities, and that linear interpolation is better for short gaps. Against the training-free HMM baseline (Table 1), DriftTraj is better in three cities and on par in Beijing.

  2. The method scaling with model parameters (Reviewer tfFy). A generator with twice the parameters (8.37M to 16.59M), trained on the same Porto data, improves Recall, MAE, and RMSE in all four cities under both Mean@5 and Top@5, including the three cities it never saw. Its validation error stays below that of the original model throughout training (App. M, Tables 22 and 23, Figure 6).

  3. Road knowledge and visual generation are complementary (Reviewers tfFy and Rmp9). We explain why road-level Recall is lower (Sec. 5.2) and add a simple fusion with a road-network route that needs no ground truth (App. K, Table 19). The fused result exceeds the best baseline value on every metric in every city, which also answers how the five candidates can be used without an oracle.

  4. Additional baselines (Reviewers C1S2 and Rmp9). Besides HMM, we add a drifting model that predicts coordinates directly from the same visual input (App. G, Table 12). DriftTraj has lower MAE and RMSE in all four cities.

  5. Robustness to rendering (Reviewers C1S2 and tfFy). Changing the zoom level or the basemap style degrades the results moderately (App. A.1, Table 6), so the revision asks for a fixed rendering protocol. We also report the meters-per-pixel statistics of the maps (App. A.1, Table 5).

  6. Non-vehicular data (Reviewer tfFy). Zero-shot results on GeoLife walking and biking trajectories are given in App. L.

We hope these additions resolve the main concerns, and we are glad to provide further details during the discussion.

AuthorsOpenReview ↗

Summary of the Rebuttal Stage

Dear Area Chair, Senior Area Chairs, and Program Chairs,

As the discussion period is coming to an end, we would like to briefly summarize the exchange with the reviewers.

First, all three reviewers found the core idea new: using rendered maps as a visual representation shared across cities, so that a model trained in one city can recover trajectories in another without learning a city-specific road vocabulary. Reviewers tfFy and Rmp9 also pointed to the marker-aware geometric supervision as a concrete solution that makes the video-based pipeline usable. Reviewers C1S2 and tfFy considered the evaluation comprehensive, covering four cities, zero-shot transfer, and extensive ablations. Reviewer tfFy further highlighted the one-step generation that makes visual recovery practical.

After the rebuttal, Reviewer C1S2 raised the score to Weak Acceptance, and Reviewer tfFy raised it to Clear Acceptance, stating that the responses addressed all of their concerns. Reviewer Rmp9 acknowledged that the HMM comparison supports our claim, and that the questions on metrics and post-processing are resolved. Rmp9 then raised three follow-up questions on runtime, controls for the fusion experiment, and route complexity. We answered all three with new experiments while not yet received a reply.

We sincerely thank all the reviewers for their careful and constructive feedback, and the Area Chair and Program Chairs for their time in organizing the discussion. The reviewers' suggestions led to a substantially stronger paper.

Best regards,

Authors of Paper 16248

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.