- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work introduces CityRep, a unified benchmark for urban representation learning that mitigates spatial leakage via block-based train-test splits. It integrates multi-source urban data across eight cities and eight downstream tasks, and systematically evaluates eleven existing representation methods. Experiments reveal that random splits overestimate model performance, while globally pretrained geospatial representations demonstrate stronger cross-city transfer of downstream predictors.
Critical strengths
The paper introduces a comprehensive benchmark covering multiple cities, tasks, and representation modalities, providing a unified framework for evaluating heterogeneous urban representations.
Critical weaknesses
- The benchmark evaluates models with substantially different pretraining scales and data sources, making it difficult to determine whether performance differences reflect representation quality or advantages in data availability and pretraining.
- The spatial alignment strategies differ across representation types, potentially introducing systematic advantages or disadvantages.
- Several downstream targets are derived from remote-sensing or geospatial products, which may favor representations pretrained on similar data sources. This raises concerns about whether the benchmark measures general urban understanding or alignment with specific data modalities.
- The cross-city transfer evaluation applies source-city prediction heads directly to target-city embeddings, which may not share aligned representation spaces for city-specific models. Consequently, the results may reflect embedding-space incompatibility rather than genuine differences in transferability.
- I strongly recommend that the authors redesign Figure 1, as its font sizes and element proportions are inconsistent, and the image resolution is low. The figure also appears AI-generated and would benefit from a more professional and carefully structured layout.
Questions for authors
- Could the authors further investigate how different spatial alignment strategies affect model performance and rankings, to ensure fair comparisons across representation types?
- Could the authors evaluate whether downstream tasks derived from remote-sensing or geospatial products systematically favor models pretrained on similar data sources, and clarify how this potential bias affects the benchmark conclusions?
- Will the authors consider releasing the data preparation pipelines and metadata for the different urban representation methods, particularly for street-view and satellite imagery? These resources appear to be missing from the current repository and would improve reproducibility.
I would be willing to raise my score if the authors can address my concerns.
3 comments
Response to Reviewer KXg4 (part 2/2).
W3/Q2: Labels derived from remote-sensing products. We grouped the tasks by how their labels are produced and recomputed the ranking within each group (App. E.4, Table 19). Mean lead is the best score of AETHER, AlphaEarth and TESSERA minus the best score of the other models (R²; F1 for LUC; AGE omitted).
| Group (tasks) | AETHER | AlphaEarth | TESSERA | SatCLIP | τ vs all tasks | Mean lead |
|---|---|---|---|---|---|---|
| Remote sensing (NTL, LST) | 2 | 1 | 3 | 11 | 0.86 | 0.15 |
| Remote sensing + other sources (PM2.5, POP, GDP) | 1 | 2 | 3 | 11 | 0.81 | 0.29 |
| Not remote sensing (RDE, LUC, AGE) | 1 | 2 | 3 | 11 | 0.81 | 0.27 |
AETHER, AlphaEarth and TESSERA occupy the top three positions in all three label-source groups, although AETHER and AlphaEarth exchange first and second place on remote-sensing labels. Their mean lead on labels not derived from remote sensing is 0.27, compared with 0.15 on remote-sensing labels and 0.29 on mixed-source labels. SatCLIP (Sentinel-2) ranks last in every group. These results suggest that the leading group’s advantage is not solely driven by remote-sensing-derived labels. This grouped analysis does not rule out modality-related bias in individual tasks.
W4: Embedding-space incompatibility in transfer. In the paper, the seven per-city models are trained separately for each city, following their original papers, so that the benchmark evaluates these methods as they were proposed. We agree, however, that the reviewer's point is well taken. Cross-city transfer lies outside the setting that several of these methods were designed for, and separate per-city training gives each city its own embedding space.
We therefore adapted the methods to this setting. We retrained each of these seven methods once on the pooled data of all 8 cities, so that each method has a single shared embedding space, with the same pipeline and inputs as its per-city version. We then re-ran the cross-city protocol (Sec. 4.5, Table 5). Rows in italics are unchanged from the separately trained setting (Table 13 in App. D.2).
| Model | LUC↑ | RDE | POP | AGE↓ | GDP | NTL | PM2.5 | LST | Mean r |
|---|---|---|---|---|---|---|---|---|---|
| AETHER | .15 | .60 | .63 | .060 | .70 | .72 | .31 | .52 | .58 |
| AlphaEarth | .15 | .59 | .50 | .125 | .65 | .74 | .19 | .56 | .54 |
| TESSERA | .16 | .64 | .59 | .091 | .65 | .71 | .19 | .39 | .53 |
| MuseCL | .10 | .47 | .48 | .059 | .55 | .62 | .28 | .40 | .47 |
| CityFM | .11 | .47 | .48 | .063 | .55 | .59 | .27 | .26 | .44 |
| Place2Vec | .12 | .45 | .44 | .056 | .47 | .56 | .23 | .38 | .42 |
| Urban2Vec | .10 | .42 | .41 | .064 | .48 | .58 | .18 | .26 | .39 |
| CaLLiPer | .07 | .27 | .37 | .056 | .43 | .47 | .12 | .24 | .32 |
| Space2Vec | .06 | .25 | .31 | .058 | .41 | .46 | .17 | .27 | .31 |
| PE | .04 | .05 | −.03 | .083 | −.04 | .08 | .15 | .11 | .05 |
| SatCLIP | .02 | −.03 | −.06 | .400 | −.08 | −.09 | .02 | .05 | −.03 |
With a shared embedding space, all seven methods transfer much better than with separate training (Table 13). The broadly pretrained raster representations (AETHER, AlphaEarth, TESSERA) still rank top 3, and the coordinate-based encoders still rank lowest. We have revised the interpretation in Sec. 4.5 accordingly.
W5: Figure 1. We have taken this suggestion and redrawn Figure 1 from scratch as a vector graphic, with a simplified layout and all labels at least 6 pt at print size. The new figure is included in the revised PDF.
Q3: Data-preparation pipelines and metadata. As requested, we have released the data-preparation pipelines and metadata for the different urban representation methods in the code repository: data_preparation/ (download and preparation of the raw inputs, including street-view and satellite imagery) and metadata/inputs/<city>/ (per-city input metadata). We will continue to improve these resources.
We hope these responses address the reviewer's concerns.
Dear Reviewer KXg4, as the rebuttal period is coming to an end, we would really appreciate hearing your thoughts and having the opportunity for further discussion.
We have carefully addressed all your concerns, including additional analyses of data coverage and spatial alignment sensitivity, an investigation of potential modality bias, a fairer cross-city transfer evaluation, and redesign of figure 1, among other improvements. The additional experiments consistently support our main conclusions and have helped us make the paper more rigorous and robust.
We look forward to hearing from you and wish you all the best in your research and beyond. We sincerely appreciate your constructive suggestions, and thank you again for your time and consideration.
We thank the reviewer for the constructive comments, which have been very helpful in improving the quality of the paper. We have tried to address every point, and our detailed responses follow (part 1/2).
W1: Data and pretraining scale. We agree that model performance depends on data and training scope. We regard this dependence as part of a model's practical usability, since sparse inputs hurt results and better pretraining data usually help. A benchmark of released models cannot fully separate the two.
To further quantify the effect of data availability, we treat a task unit as covered by a model if its aligned feature is non-zero and report the coverage of every model in every city (App. C.3, Table 11). For each city–task, we then re-scored all models only on the test units covered by all 11 models (App. C.3, Table 12). The trained heads are unchanged. Mean coverage is averaged over all city–task pairs.
The ranking is largely unchanged (Kendall τ = 0.91; the top 6 positions are identical).
W2/Q1: Alignment strategies. Following the reviewer's suggestion, we evaluated every model under two alignment groups (Sec. 4.4, Tables 3 and 4; full results in App. E.3, Table 18):
All 11 models are ranked under each group. Mean R² is averaged over the six regression tasks.
Alignment changes individual scores, and Group A gives higher regression R² for almost every model. Under both groups the top 5 are identical, and only positions 6–10 shift, by at most two places. Kendall τ is 0.93 between A and B. These results support the robustness of the main conclusions to the two tested alignment settings.