acceptodds
Under review as a conference paper at ICLR 2027

TRAJGANR: Trajectory-Centric Urban Multimodal Learning via Geospatially Aligned Neural Representations

Abstract

Many urban prediction tasks depend not only on the static attributes of a location, such as the visual appearance of its built environment, but also on its dynamic function: how people navigate and use it over time. Predicting congestion, travel demand, or road safety risks, for instance, requires mobility information that imagery and geographic coordinates alone cannot provide. Trajectories carry this signal, yet current trajectory representations are difficult to align with static geospatial observations: sparse, irregular GPS samples rarely coincide with street-view image coordinates, and existing encoders produce either a single embedding per trajectory or road segment, or embeddings only at the observed GPS samples. We introduce TrajGANR, a trajectory-centric multimodal self-supervised learning framework that represents each trajectory as a continuous, path-conditioned neural field. Reconstructed from the observed GPS samples, this representation can be queried at arbitrary coordinates (including the off-path locations where street-view images were captured), yielding localized mobility embeddings that align with street-view image and location representations at the same coordinate. Across four fine-grained road-level tasks in San Francisco and Porto, TrajGANR consistently outperforms recent geospatial and trajectory foundation models, with ablations attributing the gains to both the mobility signal and the fine granularity at which it is aligned. TrajGANR's location encoder further generalizes beyond the road network to areas where neither street-view imagery nor trajectories are available.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
multimodal self supervised learning (MSSL), human mobility trajectories, trajectory-centric geospatial MSSL

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yt6jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors tackle the problem that GPS samples are often sparse and rarely coincide with street-view image coordinates, making it diffucult to align dynamic trajectory information with static geospatial representations. They then propose TrajGANR as a multimodal self-supervised learning framwork that treat each trajectory as a neural field, allowing localized mobility embeddings to be queried at arbitrary coordinates through contrastive learning among three different modalities.

Tested on four road-segment-level tasks across two city areas, TrajGANR outperforms image-based geospatial pretraining models and a trajectory foundation model. The authors then prove the TrajGANR's ability to generalize outside the obersved trajectories through a single POI popularity prediction task.

Critical strengths
  • The paper is well motivated: there's often a mismatch between the locations where vehicles' GPS samples are collected and those where street-view images are captured. While it's widely recognized that human mobility is valuable in geospatial pretraining, the authors claim that there's no existing framework that aligns GPS trajectories with static geospatial representations.
  • The model's generalizability to off-network locations is tested against existing pretrained location encoders and shows improved performance.
  • The methodology is clearly described.
Critical weaknesses
  • The paper frames its contribution around the form vs. function of urban space. However, with trajectories at road level, the learned signal seems to reflect traffic behaviour rather than function (how people use places), and the downstream tasks are mostly traffic-related. Moreover, prior work has already aligned mobility data with other urban modalities, with a similar form-vs-function motivation [1,2,3]. The paper should at least position itself conceptually against this line of work.
  • The paper emphrasizes on the value of fine granularity a lot while all evaluation downstream labels are defined at the road-segment level, so the benefit seems not directly evaluated. Can authors include some tasks at fine-grained levels?
  • The method assigns a synthetic timestamp to the query location. which is essentially linear interpolation. It is therefore unclear how much the proposed "neural implicit field" adds beyond simple interpolation, e.g., linearly interpolating the embeddings of the neighboring GPS points. Please justify this concept and show that the gain comes from contextual modeling rather than interpolation alone.
  • The authors state that evaluation locations and street-view images are unseen during fine-tuning, but were they seen during pretraining? If so, the setting is transductive and results may be inflated by spatial autocorrelation. Please clarify.

[1] Wen, Ya, et al. "MoRA: Mobility as the backbone for geospatial representation learning at scale." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Zhou, Silin, et al. "Heterogeneous region embedding with prompt learning." Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 4. 2023.

[3] Li, Zechen, et al. "Urban region embedding via multi-view contrastive prediction." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 8. 2024.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer yt6j (1/2)

W1a “The paper frames its contribution around the form vs. function of urban space. However, with trajectories at road level, the learned signal seems to reflect traffic behaviour rather than function (how people use places), and the downstream tasks are mostly traffic-related.”

Answer. TrajGANR encodes how people navigate the city and use urban space, not traffic conditions alone. This is because it aligns each individual, whole trajectory with every location it passes, so a location's representation reflects where those trips start and end, which neighborhoods they connect, when they happen, and whether vehicles pass by or stop there. Our tasks directly reflect this: traffic speed and hard braking events test driving behavior, while the other three, road popularity (navigation demand to destinations along a road), AOI function (the surrounding mix of POI categories), and POI popularity (visit demand to individual venues), measure how places are used. Trajectories improve all three place-use tasks: adding them raises road popularity F1 from 0.921 to 0.960 and reduces AOI function error by 54% (Table 4), and pretraining with them improves POI popularity F1 by 28% over GAIR (Table 5).

W1b “Moreover, prior work has already aligned mobility data with other urban modalities, with a similar form-vs-function motivation [1,2,3]. The paper should at least position itself conceptually against this line of work.”

Answer. While [1–3] share the motivation that human movement reveals urban function, they model macro region-level socioeconomic patterns from aggregated origin-destination (OD) flows, whereas TrajGANR models micro coordinate-level representations from raw GPS trajectories. We have revised the related work to position TrajGANR within this line of work: HREP [2] and ReCP [3] partition a city into zones (such as Census tracts) and represent movement as OD trip counts, that is, how many trips start in zone i and end in zone j, together with POI data. ReCP, for instance, starts from taxi trips but keeps only these counts, discarding the route taken in between. MoRA [1] builds a flow graph between coarse H3 cells (~36 km² each) and aligns it with POIs, demographics, and satellite imagery. In all three, every modality is aggregated to the same zone or cell, so different modalities are co-located by construction and can be aligned directly using standard contrastive learning techniques. This alignment is not possible with raw GPS trajectories: a trajectory is a continuous path rather than a property of a fixed unit, and its sparse GPS samples rarely coincide with the locations where street-view images are taken. TrajGANR addresses this by representing each trajectory as a continuous function queried at exact image coordinates, which yields micro, coordinate-level representations that link how a specific location is driven to what it looks like.

W2 “The paper emphrasizes on the value of fine granularity a lot while all evaluation downstream labels are defined at the road-segment level, so the benefit seems not directly evaluated. Can authors include some tasks at fine-grained levels?”

Answer. The granularity of prediction labels need not determine the granularity of representation learning. To illustrate this, we added two new visualizations to the revised paper. The first (Fig. 3) shows that conditions vary within a segment: along a single 180 m segment, movement contexts change from a busy intersection toward a residential neighborhood. Segment-based alignment assigns every coordinate the same trajectory embedding, obscuring this variation. Our granularity ablation confirms its practical importance: coordinate-level alignment reduces prediction error by 43% for traffic speed and 60% for hard braking events (Table 3), despite segment-level labels. We also extended the individual-POI popularity evaluation (where the labels are per POI coordinate) to include the segment-based variant. Using the pretrained location encoder from coordinates alone, coordinate-level alignment achieves F1 = 0.317 vs 0.304 (Table 5). The second visualization (Fig. 4) illustrates how coordinate-level alignment helps preserve distinctions between neighboring POIs that segment-based alignment collapses.

AuthorsOpenReview ↗

Response to Reviewer yt6j (2/2)

W3 “The method assigns a synthetic timestamp to the query location. which is essentially linear interpolation. It is therefore unclear how much the proposed "neural implicit field" adds beyond simple interpolation, e.g., linearly interpolating the embeddings of the neighboring GPS points. Please justify this concept and show that the gain comes from contextual modeling rather than interpolation alone.”

Answer. Only the query's timestamp is interpolated, not its location or its representation. The query is placed at the street-view image's own coordinate, which often lies 20–50 m off the GPS path, and its embedding is computed by self-attention over the entire trajectory, so each query is contextualized by the full trip rather than by its two neighboring points. To test whether the gain depends on interpolation, our paper includes a masked-time variant (Table 7) that removes the interpolated timestamp entirely and replaces it with a learned token, leaving the model to infer the query's position in the trip from context alone. This variant performs close to the full model. Table 5 further shows that what TrajGANR learns goes beyond interpolation: its location encoder represents locations from coordinates alone, with no trajectory available at inference and therefore no neighboring GPS points to interpolate, and still outperforms all baselines. The gain therefore comes from contextual modeling rather than from interpolation.

W4 “The authors state that evaluation locations and street-view images are unseen during fine-tuning, but were they seen during pretraining? If so, the setting is transductive and results may be inflated by spatial autocorrelation. Please clarify.”

Answer. Our spatial split over the road network applies to both pretraining and fine-tuning. We have revised the Appendix to state this explicitly. In addition, the POI popularity task in Table 5 evaluates locations away from the road network, which were not part of pretraining at all.

Thank you for your response. Figures 3 and 4 provide useful illustrations of within-segment variation and the potential benefits of fine-grained alignment. However, I still see limited contributions of trajectory-centric fine-grained learning in broader non-traffic scenarios. Therefore, I maintain my original rating.

We thank the reviewer for the follow-up. We are glad Figures 3 and 4 were helpful. We emphasize that our core contribution is the alignment of raw GPS trajectories with street-view images and locations in multimodal self-supervised learning, which no prior work does. Both TrajGANR (Segment-based) and TrajGANR (Fine-grain) are instances of this contribution, and both clearly outperform existing segment-level trajectory models and established GeoFMs on diverse urban tasks including static tasks (e.g, on San Francisco AOI function, both reduce error by 66% over the strongest existing segment-level model). Fine-grained alignment then adds further gains where within-segment variation matters, on traffic speed, hard braking, and point-level POI popularity. Could the reviewer specify which concern about the non-traffic evaluation remains unaddressed?

Reviewer_E6nsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper focuses on learning representations for geographical locations, and proposes TrajGANR that combines static attributes of a location with dynamic information extracted from GPS trajectories. The proposed method treats each trajectory as a continuous function that can return an embedding at any coordinate near the trajectory's path. A Transformer reads the trajectory's GPS points together with extra "query" entries at the coordinates of nearby street-view photos, and outputs a local mobility embedding at each query. During training, three contrastive learning losses are used to align the embeddings of multiple sources. The evaluation uses public taxi GPS data from San Francisco and Porto with Mapillary street-view photos. The proposed method has the best mean value on every reported metric in both cities.

Critical strengths
  1. The evaluation covers four road-segment tasks in San Francisco and three in Porto, each scored with two metrics, plus an off-network POI-popularity task.
  2. The proposed method has the best mean value across several datasets, tasks, and metrics.
Critical weaknesses
  1. UniTraj is the only trajectory encoder compared, while methods like UniTR and JGRM are discussed but not compared in the experiments.
  2. The claim that the proposed method is "the first framework to incorporate continuous trajectories into geospatial multimodal pretraining" and that "no existing MSSL framework operationalizes this by aligning GPS trajectories with static urban modalities" can be debatable. On paper, works like MuseCL [1] and MM-Path [2] also fit into the category of "combining mobility data with imagery in contrastive or multimodal training".

[1] Yong, Xixian, and Xiao Zhou. "Musecl: predicting urban socioeconomic indicators via multi-semantic contrastive learning."

[2] Xu, Ronghui, et al. "Mm-path: Multi-modal, multi-granularity path representation learning."

Questions for authors
  1. Could the authors reiterate the uniqueness of the proposed method, compared to existing methods that share the idea of aligning mobility data with imagery modalities?
3 comments
AuthorsOpenReview ↗

Response to Reviewer E6ns (1/2)

W1. “UniTraj is the only trajectory encoder compared, while methods like UniTR and JGRM are discussed but not compared in the experiments.”

Answer. Per your suggestion we have added both UniTR [3] and JGRM [4] as segment-based trajectory baselines in Tables 1-2. Both learn one embedding per road segment from map-matched trajectories: UniTR uses the sequence of road segments each trip traverses with street view images and other road level features, while JGRM additionally encodes the speed and acceleration of GPS points within each segment. We pretrain both on the same trajectories and spatial splits as TrajGANR and evaluate them with the same protocol and task heads. TrajGANR outperforms both on every task in both cities:

SF Speed (MAE↓)SF Popularity (F1↑)SF AOI (L1↓)SF HBE (MAE↓)Porto Popularity (F1↑)Porto AOI (L1↓)Porto HBE (MAE↓)
UniTR0.6150.9220.3591.8710.7710.3828.310
JGRM0.6880.9190.4040.6750.7700.4173.709
TrajGANR0.4030.9600.1210.1300.9080.2790.401

Both baselines outperform UniTraj but remain substantially behind TrajGANR. Like UniTraj, they assign a single embedding to each road segment, collapsing variation in driving behavior within the segment. TrajGANR instead evaluates each individual trajectory at an image’s coordinate and aligns the resulting embedding with the image and location embeddings there before aggregating across trajectories, preserving the localized movement information that segment-level alignment obscures.

W2. “The claim that the proposed method is "the first framework to incorporate continuous trajectories into geospatial multimodal pretraining" and that "no existing MSSL framework operationalizes this by aligning GPS trajectories with static urban modalities" can be debatable. On paper, works like MuseCL [1] and MM-Path [2] also fit into the category of "combining mobility data with imagery in contrastive or multimodal training".

Answer. "Mobility data" is a broad term that covers many forms, from aggregated flow counts between regions to individual GPS traces. Our work focuses specifically on GPS trajectories, as reflected in the title ("Trajectory-Centric Urban Multimodal Learning"), stated in Section 1 ("aligning GPS trajectories with static urban modalities"), and formally defined in Section 2 (Definition 2) as ordered sequences of timestamped GPS points recorded along individual trips. We have consistently maintained this interpretation throughout the paper. Moreover, neither MuseCL nor MM-Path use GPS trajectories. MuseCL summarizes large hexagonal regions by two numbers: its total inflow and outflow of people, and uses these counts only to choose which street-view images form positive pairs, so no trajectories or mobility are encoded as input. MM-Path uses no trajectories at all, but aligns road-network graph nodes with remote-sensing images.

TrajGANR's contribution is to encode raw GPS trajectories as a new input modality and align them with street-view images and coordinates at the coordinate level. This requires solving a problem that neither work faces: GPS samples rarely coincide with image locations, so there are no co-located pairs to align. TrajGANR addresses this by representing each trajectory as a continuous function that can be queried at any nearby image location, which yields the movement context at that exact point. This lets the model learn how a specific location is driven and relate it to what that location looks like, information that region-level flow counts and road graph representations do not capture.

AuthorsOpenReview ↗

Response to Reviewer E6ns (2/2)

Q1. “Could the authors reiterate the uniqueness of the proposed method, compared to existing methods that share the idea of aligning mobility data with imagery modalities?”

Answer. As detailed in our response to W2, existing methods that combine mobility with imagery use aggregated mobility, such as flow counts between regions, at a fixed spatial unit. TrajGANR differs in three ways, all of which are unique to the best of our knowledge:

  • (1) Data: it encodes raw GPS trajectories, the actual paths of individual trips, rather than aggregated counts.
  • (2) Alignment: because GPS samples rarely coincide with image locations, it represents each trajectory as a continuous function that can be queried at any image's coordinate, and aligns each trajectory with each image at that exact location.
  • (3) Result: the learned representation captures how people navigate through and use a specific location, including where trips start and end, whether vehicles pass by or stop, and how they slow, brake, or flow freely, and relates this to what the location looks like, at the level of individual coordinates rather than regions or road paths.

[1] Yong, Xixian, and Xiao Zhou. "Musecl: predicting urban socioeconomic indicators via multi-semantic contrastive learning."

[2] Xu, Ronghui, et al. "Mm-path: Multi-modal, multi-granularity path representation learning."

[3] Unitr: A unified framework for joint representation learning of trajectories and road networks. AAAI, 2025.

[4] More than routing: Joint gps and route modeling for refine trajectory representation learning. In Proceedings of the ACM Web Conference 2024.

I would like to thank the authors for their detailed response, which largely answered the concerns I raised in my original assessment. I will raise my rating accordingly.

Reviewer_XDkuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TrajGANR, a multimodal self-supervised framework that integrates GPS trajectories, street-view images, and geographic coordinates for fine-grained location representation learning. The key idea is to use coordinate-conditioned queries to infer localized trajectory embeddings at locations without observed GPS samples. Experiments on road-level prediction tasks in San Francisco and Porto show improvements over existing geospatial and trajectory representation models.

Critical strengths
  1. The paper addresses the challenge of aligning sparsely sampled GPS trajectories with static geospatial modalities. Incorporating mobility information into geospatial representation learning is a meaningful direction.

  2. The coordinate-conditioned trajectory representation is an interesting design choice. Using query tokens and joint spatiotemporal attention to infer embeddings at unobserved locations provides a simple and coherent solution to the spatial alignment problem.

  3. The experimental results are strong across two cities and multiple downstream tasks. The modality and alignment-granularity ablations further support the effectiveness of incorporating trajectory information and fine-grained alignment.

Critical weaknesses
  1. The need for fine-grained trajectory representations is not clearly motivated. The paper proposes learning trajectory embeddings at arbitrary geographic coordinates, but does not explain why such fine spatial granularity is necessary. Moreover, all four downstream tasks use road-segment-level labels, further raising the question of whether coordinate-level representations are needed.

  2. The proposed framework mainly combines existing techniques, including Space2Vec, Time2Vec, Transformer-based query encoding, and contrastive learning. While the overall design is reasonable, the technical contribution beyond adapting these components for trajectory–image alignment appears relatively incremental.

  3. The benefits of fine-grained alignment are not convincingly isolated. The alignment ablation compares the proposed coordinate-level querying mechanism with mean-pooled road-segment embeddings. However, these methods differ in both representation granularity and aggregation strategy. Comparisons with stronger segment-level trajectory encoders are needed to establish whether the improvements specifically come from fine-grained alignment.

Questions for authors

Please refer to the weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer XDku (1/2)

W1a. "The need for fine-grained trajectory representations is not clearly motivated. The paper proposes learning trajectory embeddings at arbitrary geographic coordinates, but does not explain why such fine spatial granularity is necessary"

Answer: Our GPS trajectories are sparse, and we do not assume finely sampled observations. This sparsity is the motivation: GPS samples rarely coincide with the locations where street-view images are taken, so directly matched pairs are unavailable. We represent each trajectory as a continuous neural implicit function that can be evaluated at an image's coordinate, allowing each individual trajectory to align with the image and location embeddings there before aggregation across trajectories. As a result, the model can also represent any coordinate, not only observed GPS points or road segments.

W1b. “Moreover, all four downstream tasks use road-segment-level labels, further raising the question of whether coordinate-level representations are needed.”

Answer: The granularity of prediction labels need not determine the granularity of representation learning. To illustrate this, we added two new visualizations to the revised paper. The first (Fig. 3) shows that conditions vary within a segment: along a single 180 m segment, movement contexts change from a busy intersection toward a residential neighborhood. Segment-based alignment assigns every coordinate the same trajectory embedding, obscuring this variation. Our granularity ablation confirms its practical importance: coordinate-level alignment reduces prediction error by 43% for traffic speed and 60% for hard braking events (Table 3), despite segment-level labels. We also extended the individual-POI popularity evaluation (where the labels are per POI coordinate) to include the segment-based variant. Using the pretrained location encoder from coordinates alone, coordinate-level alignment achieves F1 = 0.317 vs 0.304 (Table 5). The second visualization (Fig. 4) illustrates how coordinate-level alignment helps preserve distinctions between neighboring POIs that segment-based alignment collapses.

W2. “The proposed framework mainly combines existing techniques, including Space2Vec, Time2Vec, Transformer-based query encoding, and contrastive learning. While the overall design is reasonable, the technical contribution beyond adapting these components for trajectory–image alignment appears relatively incremental.”

Answer: Space2Vec, Time2Vec, Transformer encoders, and contrastive learning are established building blocks, and we do not claim novelty for these components individually. Our contributions concern the information incorporated into geospatial representations, how the modalities are aligned, and what this alignment enables. Specifically, (1) we introduce raw GPS trajectories as a dynamic, movement-based modality for multimodal geospatial representation learning. (2) We develop a new cross-modal contrastive self-supervised objective to fuse this modality with the other inputs. Existing geospatial models align modalities observed at the same locations, but sparse GPS samples rarely coincide with image locations, so no such pairs exist. We address this by representing each trajectory as a continuous neural implicit function that can be queried at any image's coordinate, and by aligning each individual trajectory's embedding at that coordinate with the image and location embeddings there, before any aggregation across trajectories. (3) We demonstrate that incorporating movement information improves even static downstream tasks: in our modality ablation (Table 4), adding trajectories cuts AOI function error by 54%, although that task predicts the surrounding POI-category distribution rather than traffic. Trajectories can describe how people navigate and how places are connected, complementing the appearance and physical structure captured by imagery. Our results prove that this dynamic information benefits representations beyond tasks explicitly concerned with movement.

AuthorsOpenReview ↗

Response to Reviewer XDku (2/2)

W3. “The benefits of fine-grained alignment are not convincingly isolated. The alignment ablation compares the proposed coordinate-level querying mechanism with mean-pooled road-segment embeddings. However, these methods differ in both representation granularity and aggregation strategy. Comparisons with stronger segment-level trajectory encoders are needed to establish whether the improvements specifically come from fine-grained alignment.”

Answer: We agree with the reviewer that TrajGANR (Segment-based) and TrajGANR (Fine-grain) variants differ in aggregation and granularity. This difference is unavoidable for any segment-level representation, since each segment must be summarized into a single vector, whether by mean-pooling or by a learned encoder. Per your suggestion, we tested whether a stronger segment encoder could close the gap by adding two recent road-segment trajectory encoders, UniTR [1] and JGRM [2], to Tables 1–2 in the revised paper. Below we show a summary between the two variants and the new baselines for San Francisco with respect to MAE ↓:

MethodRepresentationTraffic speedHBE
UniTROne vector per segment0.6151.871
JGRMOne vector per segment0.6880.675
TrajGANR (Segment-based)One vector per segment0.7090.328
TrajGANR (Fine-grain)Queried at each coordinate0.4030.130

The more expressive segment encoders perform similarly to our mean-pooled variant, slightly better on traffic speed and worse on HBEs, and all three remain far behind TrajGANR (Fine-grain). Conclusion: A better segment summary does not close the gap, which confirms that the improvement comes from fine-grained alignment.

[1] Unitr: A unified framework for joint representation learning of trajectories and road networks. AAAI, 2025.

[2] More than routing: Joint gps and route modeling for refine trajectory representation learning. In Proceedings of the ACM Web Conference 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.