acceptodds
Under review as a conference paper at ICLR 2027

CITYREP: A Unified Benchmark for Urban Representations Across Cities, Tasks, and Modalities

40 views

Abstract

Urban representation learning encodes complex urban environments into general-purpose embeddings for diverse downstream tasks and emerging urban foundation models. However, current evaluations are limited, typically focusing on one or two cities and tasks and relying on random splits that introduce spatial leakage, leading to inflated performance and weak support for cross-location generalization and fair comparison. To address this, we propose CityRep, a unified benchmark that evaluates urban representations across data modalities, cities, and tasks using spatially structured splits. CityRep consists of three key components: (1) a spatial unit-agnostic evaluation framework that supports heterogeneous urban representations through a standardized alignment module; (2) a unified evaluation protocol using block-based spatial splits to mitigate spatial leakage and enable rigorous model comparison; and (3) an extensible multi-city, multi-task benchmark suite spanning 8 cities and 8 tasks across regression, classification, and distribution prediction. We evaluate 11 representative urban representation models. Results show that performance is highly sensitive to the split protocol, with random splits inflating scores and altering model rankings. We also observe substantial variability across cities and tasks, underscoring the need for generalization-aware evaluation. CityRep is released as a reproducible benchmark with datasets, evaluation pipelines, and diagnostic tools to facilitate fair comparison and support future research in urban representation learning towards urban foundation models.

Primary area:
datasets and benchmarks
Keywords:
Urban Representation Learning, Geospatial Benchmarking, Multimodal Learning, Urban Foundation Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KXg4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces CityRep, a unified benchmark for urban representation learning that mitigates spatial leakage via block-based train-test splits. It integrates multi-source urban data across eight cities and eight downstream tasks, and systematically evaluates eleven existing representation methods. Experiments reveal that random splits overestimate model performance, while globally pretrained geospatial representations demonstrate stronger cross-city transfer of downstream predictors.

Critical strengths

The paper introduces a comprehensive benchmark covering multiple cities, tasks, and representation modalities, providing a unified framework for evaluating heterogeneous urban representations.

Critical weaknesses
  1. The benchmark evaluates models with substantially different pretraining scales and data sources, making it difficult to determine whether performance differences reflect representation quality or advantages in data availability and pretraining.
  2. The spatial alignment strategies differ across representation types, potentially introducing systematic advantages or disadvantages.
  3. Several downstream targets are derived from remote-sensing or geospatial products, which may favor representations pretrained on similar data sources. This raises concerns about whether the benchmark measures general urban understanding or alignment with specific data modalities.
  4. The cross-city transfer evaluation applies source-city prediction heads directly to target-city embeddings, which may not share aligned representation spaces for city-specific models. Consequently, the results may reflect embedding-space incompatibility rather than genuine differences in transferability.
  5. I strongly recommend that the authors redesign Figure 1, as its font sizes and element proportions are inconsistent, and the image resolution is low. The figure also appears AI-generated and would benefit from a more professional and carefully structured layout.
Questions for authors
  1. Could the authors further investigate how different spatial alignment strategies affect model performance and rankings, to ensure fair comparisons across representation types?
  2. Could the authors evaluate whether downstream tasks derived from remote-sensing or geospatial products systematically favor models pretrained on similar data sources, and clarify how this potential bias affects the benchmark conclusions?
  3. Will the authors consider releasing the data preparation pipelines and metadata for the different urban representation methods, particularly for street-view and satellite imagery? These resources appear to be missing from the current repository and would improve reproducibility.

I would be willing to raise my score if the authors can address my concerns.

3 comments

We thank the reviewer for the constructive comments, which have been very helpful in improving the quality of the paper. We have tried to address every point, and our detailed responses follow (part 1/2).

W1: Data and pretraining scale. We agree that model performance depends on data and training scope. We regard this dependence as part of a model's practical usability, since sparse inputs hurt results and better pretraining data usually help. A benchmark of released models cannot fully separate the two.

To further quantify the effect of data availability, we treat a task unit as covered by a model if its aligned feature is non-zero and report the coverage of every model in every city (App. C.3, Table 11). For each city–task, we then re-scored all models only on the test units covered by all 11 models (App. C.3, Table 12). The trained heads are unchanged. Mean coverage is averaged over all city–task pairs.

ModelMean coveragePaper rankCommon-coverage rank
AETHER1.001.750 (1)1.875 (1)
AlphaEarth1.002.125 (2)2.375 (2)
TESSERA0.973.125 (3)3.250 (3)
CaLLiPer1.004.750 (4)4.625 (4)
CityFM0.826.125 (5)5.875 (5)
MuseCL0.857.125 (6)7.125 (6)
PE1.007.250 (7)7.125 (6)
Place2Vec0.857.375 (8)7.875 (10)
Space2Vec1.007.500 (9)7.375 (8)
Urban2Vec0.887.875 (10)7.375 (8)
SatCLIP1.0010.375 (11)10.375 (11)

The ranking is largely unchanged (Kendall τ = 0.91; the top 6 positions are identical).

W2/Q1: Alignment strategies. Following the reviewer's suggestion, we evaluated every model under two alignment groups (Sec. 4.4, Tables 3 and 4; full results in App. E.3, Table 18):

  • Group A (richer). Each task unit is represented with more of the model's information, by averaging multiple pixels, H3 cells, entities or query points.
  • Group B (sparser). Each unit is represented with less information, using a single location or an aggregation that keeps less of the signal.
Type (models)Group A (richer)Group B (sparser)
Raster (AETHER, AlphaEarth, TESSERA)mean of pixels in the unitmax of pixels in the unit
Region (Urban2Vec, MuseCL)area-weighted mean of overlapping H3 cellsH3 cell at the unit centroid
Sparse entity (CityFM, Place2Vec)entities averaged per H3 cell, then cell lookupmean of the k nearest entities
Coordinate (CaLLiPer, Space2Vec, PE, SatCLIP)mean of 4×4 queries in the unitquery at the unit centre

All 11 models are ranked under each group. Mean R² is averaged over the six regression tasks.

ModelMean R² (A)Mean R² (B)Rank (A)Rank (B)
AETHER0.6020.5281.750 (1)2.000 (1)
AlphaEarth0.5760.5262.250 (2)2.250 (2)
TESSERA0.5570.4553.125 (3)3.875 (3)
CaLLiPer0.3490.3094.375 (4)4.500 (4)
CityFM0.3050.2986.500 (5)5.875 (5)
MuseCL0.2570.2377.000 (6)7.250 (7)
Space2Vec0.2700.2337.125 (7)7.375 (8)
PE0.2580.2607.250 (8)6.875 (6)
Place2Vec0.2000.1787.750 (9)7.500 (9)
Urban2Vec0.2470.2248.000 (10)7.750 (10)
SatCLIP0.0710.07110.375 (11)10.500 (11)

Alignment changes individual scores, and Group A gives higher regression R² for almost every model. Under both groups the top 5 are identical, and only positions 6–10 shift, by at most two places. Kendall τ is 0.93 between A and B. These results support the robustness of the main conclusions to the two tested alignment settings.

Response to Reviewer KXg4 (part 2/2).

W3/Q2: Labels derived from remote-sensing products. We grouped the tasks by how their labels are produced and recomputed the ranking within each group (App. E.4, Table 19). Mean lead is the best score of AETHER, AlphaEarth and TESSERA minus the best score of the other models (R²; F1 for LUC; AGE omitted).

Group (tasks)AETHERAlphaEarthTESSERASatCLIPτ vs all tasksMean lead
Remote sensing (NTL, LST)213110.860.15
Remote sensing + other sources (PM2.5, POP, GDP)123110.810.29
Not remote sensing (RDE, LUC, AGE)123110.810.27

AETHER, AlphaEarth and TESSERA occupy the top three positions in all three label-source groups, although AETHER and AlphaEarth exchange first and second place on remote-sensing labels. Their mean lead on labels not derived from remote sensing is 0.27, compared with 0.15 on remote-sensing labels and 0.29 on mixed-source labels. SatCLIP (Sentinel-2) ranks last in every group. These results suggest that the leading group’s advantage is not solely driven by remote-sensing-derived labels. This grouped analysis does not rule out modality-related bias in individual tasks.

W4: Embedding-space incompatibility in transfer. In the paper, the seven per-city models are trained separately for each city, following their original papers, so that the benchmark evaluates these methods as they were proposed. We agree, however, that the reviewer's point is well taken. Cross-city transfer lies outside the setting that several of these methods were designed for, and separate per-city training gives each city its own embedding space.

We therefore adapted the methods to this setting. We retrained each of these seven methods once on the pooled data of all 8 cities, so that each method has a single shared embedding space, with the same pipeline and inputs as its per-city version. We then re-ran the cross-city protocol (Sec. 4.5, Table 5). Rows in italics are unchanged from the separately trained setting (Table 13 in App. D.2).

ModelLUC↑RDEPOPAGE↓GDPNTLPM2.5LSTMean r
AETHER.15.60.63.060.70.72.31.52.58
AlphaEarth.15.59.50.125.65.74.19.56.54
TESSERA.16.64.59.091.65.71.19.39.53
MuseCL.10.47.48.059.55.62.28.40.47
CityFM.11.47.48.063.55.59.27.26.44
Place2Vec.12.45.44.056.47.56.23.38.42
Urban2Vec.10.42.41.064.48.58.18.26.39
CaLLiPer.07.27.37.056.43.47.12.24.32
Space2Vec.06.25.31.058.41.46.17.27.31
PE.04.05−.03.083−.04.08.15.11.05
SatCLIP.02−.03−.06.400−.08−.09.02.05−.03

With a shared embedding space, all seven methods transfer much better than with separate training (Table 13). The broadly pretrained raster representations (AETHER, AlphaEarth, TESSERA) still rank top 3, and the coordinate-based encoders still rank lowest. We have revised the interpretation in Sec. 4.5 accordingly.

W5: Figure 1. We have taken this suggestion and redrawn Figure 1 from scratch as a vector graphic, with a simplified layout and all labels at least 6 pt at print size. The new figure is included in the revised PDF.

Q3: Data-preparation pipelines and metadata. As requested, we have released the data-preparation pipelines and metadata for the different urban representation methods in the code repository: data_preparation/ (download and preparation of the raw inputs, including street-view and satellite imagery) and metadata/inputs/<city>/ (per-city input metadata). We will continue to improve these resources.

We hope these responses address the reviewer's concerns.

Dear Reviewer KXg4, as the rebuttal period is coming to an end, we would really appreciate hearing your thoughts and having the opportunity for further discussion.

We have carefully addressed all your concerns, including additional analyses of data coverage and spatial alignment sensitivity, an investigation of potential modality bias, a fairer cross-city transfer evaluation, and redesign of figure 1, among other improvements. The additional experiments consistently support our main conclusions and have helped us make the paper more rigorous and robust.

We look forward to hearing from you and wish you all the best in your research and beyond. We sincerely appreciate your constructive suggestions, and thank you again for your time and consideration.

Reviewer_3sPEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

CityRep is a benchmark for evaluating urban representations across eight cities and eight downstream tasks. It provides an alignment layer for heterogeneous spatial representations and uses block-based spatial splits to reduce leakage from nearby locations. Evaluating 11 representation models, the paper finds that random splits can inflate scores and change model rankings, while results vary substantially across cities and tasks. It also reports a cross-city supervision-transfer diagnostic and releases datasets and evaluation tooling.

Critical strengths
  1. The benchmark directly addresses spatial leakage with shared spatially structured splits. The reported score inflation and rank changes under random splitting make the methodological contribution consequential.
  2. Its coverage across multiple cities, tasks, and representation modalities supports more informative comparisons than narrowly scoped evaluations, and the released pipeline should make follow-up work practical.
Critical weaknesses
  1. A single block-split design cannot represent every relevant deployment shift. Since the benchmark's conclusions about model ranking and generalization depend on the split geometry, sensitivity to alternative spatial partitions and block scales is important.
  2. Data availability and representation coverage vary across cities and modalities. This may conflate representation quality with input-data coverage and limit interpretation of the cross-city results, although the paper acknowledges this constraint.
Questions for authors
  1. How sensitive are the headline rankings and random-split inflation findings to block size and to alternative spatial partition strategies, especially for tasks with different spatial resolutions?
  2. Are representation coverage and missing-input rates reported per city and model in a way that permits separating embedding quality from data availability? This could affect interpretation of the leaderboard.
2 comments

We thank the reviewer for recognizing that CityRep directly addresses spatial leakage across multiple cities, tasks and modalities. We agree that evaluating multiple spatial partitions is valuable and have added these experiments. Our detailed responses follow.

W1/Q1: Sensitivity to split geometry and block scale. We agree that a single block design cannot represent every deployment shift. The paper already compares the main 10×10 grid with a finer 20×20 grid (App. E.2). Following the reviewer's suggestion, we have further added two partitions (Sec. 4.4, Table 3; App. E.3), which are fixed 2 km square blocks that do not depend on the city extent, and the 10×10 grid with a 1 km buffer, which removes training and validation units near the test set while keeping the test set unchanged.

Headline ranking (mean task rank, overall position in parentheses; Table 3):

ModelMain (10×10)+1 km buffer20×202 km fixedRandom†
AETHER1.75 (1)1.88 (1)2.13 (1)2.63 (1)3.63 (2)
AlphaEarth2.13 (2)2.00 (2)2.88 (2)3.25 (2)4.25 (3)
TESSERA3.13 (3)3.00 (3)3.38 (3)3.25 (2)4.25 (3)
CaLLiPer4.75 (4)4.88 (4)4.00 (4)3.88 (4)2.00 (1)
CityFM6.13 (5)7.88 (9)4.50 (5)4.13 (5)5.25 (6)
MuseCL7.13 (6)6.63 (6)8.25 (9)8.13 (9)9.75 (10)
PE7.25 (7)6.63 (6)6.38 (6)6.25 (6)7.00 (8)
Place2Vec7.38 (8)6.00 (5)8.50 (10)9.38 (10)8.88 (9)
Space2Vec7.50 (9)8.38 (10)7.13 (7)6.75 (7)4.38 (5)
Urban2Vec7.88 (10)7.63 (8)7.63 (8)7.75 (8)5.63 (7)
SatCLIP10.38 (11)10.00 (11)10.00 (11)10.13 (11)10.50 (11)
Kendall τ vs main–0.730.820.810.55

† Random splitting is shown as a reference; it is not a spatial partition.

The top 3 (AETHER, AlphaEarth, TESSERA) and the last position are unchanged under every partition. Mid-table positions are more sensitive, especially to buffering. Ordered from the strongest to the weakest spatial separation (+1 km buffer, 10×10, 20×20, 2 km fixed, random), the coordinate encoders CaLLiPer and Space2Vec rank steadily higher (mean task rank from 4.88 to 2.00 and from 8.38 to 4.38), consistent with their large gains under random splits (Sec. 4.3). They benefit from local interpolation when training and test units are close.

Random-split inflation (App. E.3, Table 17; random-split score minus spatial-split score, mean over models and cities; R² for regression, macro-F1 for LUC, −KL for AGE):

Task2 km fixed20×2010×10+1 km buffer
LUC.16.16.19.22
RDE.09.10.13.16
POP.19.19.24.30
AGE.01.02.02.02
GDP.06.10.17.22
NTL.15.15.28.35
PM2.5.07.11.24.25
LST.26.27.44.62

Inflation appears under every partition and grows with spatial separation, so it is not specific to a particular block size. Grouped by resolution, the coarse tasks (GDP, NTL and PM2.5, 0.5–1 km cells) have a mean inflation of only .09 under 2 km blocks, which contain few of their cells, rising to .27 with the buffer. The fine tasks (LUC, RDE, POP and LST, about 100 m or points) already reach .18 under 2 km blocks and .33 with the buffer.

W2/Q2: Coverage per city and model. Yes. A task unit is covered by a model if its aligned feature is non-zero; uncovered units receive zero vectors (Sec. 3.2). Coverage per city, averaged over tasks (App. C.3, Table 11):

ModelLDNNYCSGPSYDBOMNBOJKTCPT
AETHER, AlphaEarth, SatCLIP, CaLLiPer, Space2Vec, PE11111111
TESSERA1.92.871.97111
Urban2Vec.99.93.97.82.88.831.61
MuseCL.98.91.97.76.88.731.51
Place2Vec.98.91.97.76.87.731.51
CityFM.99.85.81.77.88.741.52

To assess the sensitivity of the ranking to missing coverage, we also re-scored each model's saved test predictions only on the units covered by all 11 models (51–100% of units, depending on the city) in App. C.3 (Table 12). The trained heads are unchanged. The ranking is largely unchanged (Kendall τ = 0.91 with the paper's ranking; the top 6 positions are identical). The leaderboard therefore mainly reflects representation quality rather than coverage.

We hope these responses address the reviewer's concerns.

Thank you for your response. I will maintain my score.

Reviewer_TWHhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces CityRep, a unified benchmark for evaluating urban representation learning methods across heterogeneous spatial units, multiple data modalities, diverse urban environments, and downstream tasks. The benchmark covers eight cities and eight downstream tasks, providing a relatively comprehensive evaluation framework for urban representations.

The authors conduct extensive experiments comparing a broad range of representative methods, from earlier approaches such as Urban2Vec to more recent models such as AlphaEarth. The paper also provides detailed analyses of model performance across different cities, tasks, and evaluation settings.

The experimental findings highlight several important observations, including the substantial variation in model performance across urban contexts, the importance of spatially structured data splits, and the potential advantages of broadly pretrained raster representations for cross-city generalization.

Overall, I believe the benchmark addresses a meaningful problem and could potentially serve as a useful resource for the urban representation learning community. However, the extremely poor presentation quality, together with concerns about the scope of the dataset contribution, substantially undermines the overall quality of the submission.

Critical strengths
  1. A meaningful benchmark for urban representation learning. The paper introduces a potentially valuable benchmark covering eight cities and eight downstream tasks. Such a unified evaluation framework could facilitate systematic comparisons across different urban environments and help advance research on general-purpose urban representations.

  2. Extensive comparison with existing methods. The authors evaluate a wide range of representative approaches, spanning earlier urban representation learning methods such as Urban2Vec to recent large-scale pretrained models such as AlphaEarth. This broad coverage provides a useful overview of the development and current capabilities of urban representation learning methods.

  3. Detailed experimental analysis and useful insights. The paper provides extensive analyses of model performance across different tasks, cities, and evaluation metrics. These comparisons reveal meaningful differences in model capabilities and offer potentially useful insights for future research, particularly regarding spatial generalization, cross-city transfer, and the integration of heterogeneous urban data modalities.

Critical weaknesses
  1. Extremely poor presentation quality. My primary concern is the exceptionally poor quality of the manuscript's visual presentation. Figure 1, in particular, appears to have been generated without sufficient manual refinement. Its layout and typography lack the clarity expected of a research paper, and much of the text is so small that it is difficult to read. Regardless of how the figure was produced, its current quality is unacceptable for a top-tier conference submission.

At the same time, several experimental figures are unnecessarily large and occupy a disproportionate amount of space. The inconsistent figure sizing and inefficient use of page space give the manuscript an unpolished appearance. Given the importance of clear presentation for a benchmark paper, where readers must understand and compare a large number of experimental results, I consider these issues particularly serious. The overall presentation falls substantially below the standard I would expect at ICLR and is a major factor behind my negative recommendation.

  1. Lack of vector-based figures. The figures appear to be provided exclusively in raster formats such as PNG rather than vector formats. This further compromises their readability, especially when magnified or viewed in print. For an academic paper containing numerous plots, diagrams, and textual annotations, using vector graphics such as PDF or SVG whenever appropriate is a basic presentation practice. The authors should pay considerably more attention to figure quality and consistency.

  2. Limited contribution in terms of underlying data collection. As I understand it, CityRep primarily provides a collection of downstream evaluation datasets and a unified benchmarking protocol, rather than a comprehensive dataset that includes the underlying pretraining modalities, such as satellite imagery, population information, and other large-scale urban data sources.

While assembling and standardizing downstream benchmarks is certainly valuable, such data may be relatively easier to obtain than large-scale multimodal pretraining corpora. This raises concerns about the extent of the dataset construction effort and the overall novelty of the contribution. The authors should more clearly distinguish between newly collected data, existing datasets that have been reorganized or standardized, and the evaluation protocols introduced by CityRep. A clearer discussion of the benchmark's unique contributions would help establish its significance.

Questions for authors
  1. On presentation quality: Could the authors substantially revise Figure 1 to improve the readability of its text, the consistency of its layout, and its overall visual quality? The current figure is difficult to interpret and does not meet the presentation standard expected for a top-tier conference paper.

  2. On figure formatting: Could the authors replace raster-based plots and diagrams with vector graphics wherever possible, and reconsider the dimensions of the oversized experimental figures to improve the manuscript's information density?

  3. On the dataset contribution: Could the authors clarify which components of CityRep are newly collected or constructed and which are derived from existing publicly available datasets? In particular, does CityRep provide any of the underlying pretraining data modalities, or is its contribution primarily limited to downstream evaluation datasets and standardized benchmarking protocols?

  4. On the novelty and construction effort of the benchmark: Given that downstream labels and evaluation datasets may be easier to obtain than large-scale multimodal pretraining data, could the authors elaborate on the primary technical and practical challenges involved in constructing CityRep? What distinguishes it from a straightforward aggregation and standardization of existing urban datasets?

3 comments

We sincerely thank the reviewer for recognizing CityRep as a meaningful benchmark and for the helpful comments. The reviewer's suggestions have helped us improve the presentation of the paper. Our detailed responses follow.

W1/Q1: Figure 1. We take the presentation concerns very seriously. We have redrawn Figure 1 from scratch as a vector graphic with a simplified layout and all labels at least 6 pt at print size. The new figure is included in the revised PDF.

W2/Q2: Vector graphics and figure sizes. We have converted all figures that can be vector graphics into vector PDFs. We have also redesigned Figures 2 and 3 in a wider and shorter layout with larger numbers, so that they use less space and remain legible at print size. In addition, we have placed all figures, as vector PDFs or high-resolution images, in the code repository for easier viewing.

W3/Q3: What is new and what is derived.

  • Newly constructed task datasets. LUC harmonizes land-use and zoning data from eight city-specific sources, each with its own classification scheme, into 12 shared classes (App. B.2). RDE is computed from OpenStreetMap road networks.
  • Task datasets from public products. The other six tasks (POP, AGE, GDP, NTL, PM2.5, LST) are processed from public products and aligned to common task units with stable sample ids (Table 6).
  • New embeddings and protocols. Seven of the 11 baselines (AETHER, CaLLiPer, Space2Vec, Place2Vec, CityFM, Urban2Vec, MuseCL) were trained or reproduced by us in all eight cities, which makes them comparable across cities for the first time. We release the embeddings of all 11 models. The alignment module, the spatial block splits, and the evaluation pipeline are introduced by CityRep.

Although CityRep does not provide new pretraining data, it provides ready-to-use pipelines for preparing the inputs of these models in each city, together with the released embeddings, so that multi-city models are both comparable and usable.

Q4: Novelty and construction effort. The main challenge of CityRep lies in the systematic evaluation of existing urban representation models, which requires careful experimental design and a large amount of work. Including the appendix, the paper reports about 35,000 evaluation runs, covering 11 models, 8 cities and 8 tasks under multiple split, probe and alignment settings; seven of the baselines were trained or reproduced by us in all eight cities. What distinguishes CityRep from an aggregation of existing datasets is that these experiments lead to findings that matter for the field, most notably the clear advantage of embeddings pretrained at large scale. Building on these findings, CityRep provides an easy-to-use framework for evaluating future urban representations more reliably and fairly.

We sincerely thank the reviewer again for the constructive suggestions, and we hope these responses address the reviewer's concerns.

Reviewer_TWHhOpenReview ↗

Response of the authors rebuttal

Thank you for your response.

First, regarding the overall writing and presentation quality, I believe the figures in the initial version of the paper were of extremely poor quality. They appeared to be raw AI-generated figures without any meaningful post-processing or refinement. Even in the revised version, many figures and tables are disproportionately large relative to the actual content they convey, seemingly being enlarged to fill the ten-page limit. In terms of visual presentation and overall writing quality, I do not believe this paper meets the standards expected of an ICLR submission.

Second, regarding your response about the newly constructed tasks, particularly land-use prediction, I would like to point out that this task is by no means new in the context of urban representation learning. Previous works, such as MVURE (Zhang et al., 2020) and MGFN (Wu et al., 2022), have already explored closely related downstream tasks, namely land usage classification and land usage type prediction, respectively. Therefore, I do not think it is appropriate to characterize land-use prediction as a newly constructed task in this work.

References:

  • Zhang et al., "Multi-View Joint Graph Representation Learning for Urban Region Embedding," IJCAI 2020.
  • Wu et al., "Multi-Graph Fusion Networks for Urban Region Embedding," IJCAI 2022.

Finally, although the paper benchmarks a relatively large number of baselines, I believe the overall amount of substantive research and technical contributions remains rather limited. Considering these issues, I do not find the authors' response sufficient to address my concerns, and I therefore maintain my original rating.

We thank the reviewer for the follow-up comments and would like to clarify two points.

1. Figure and table sizes. Due to the page limit, we cannot even accommodate all our experimental results and analyses in the main text. The figures and tables were not enlarged to fill space, but to ensure readability at print size, in response to the reviewer's initial concerns about readability.

2. Newly constructed LUC dataset. We fully acknowledge that LUC is an established and common downstream task in urban representation learning, which is precisely why we include it in our benchmark. The manuscript itself does not claim that LUC is a new task, and we apologize for the ambiguity introduced by the wording in our previous response. Our contribution is not a new task, but a newly constructed, ready-to-use evaluation dataset that harmonizes land-use data from eight cities.

We would like to reiterate that the primary value of CityRep lies in providing a timely and standardized benchmark that addresses an important need in the urban representation learning community. Thank you again for your time and efforts.

Comments on the paper

2 comments

We thank all reviewers for their constructive feedback and for recognizing the value of CityRep. We are encouraged that the reviewers generally acknowledged the paper’s soundness. We believe CityRep makes a timely contribution to the urban representation learning community by providing a unified, comparable benchmark across cities, tasks, and modalities, enabling more rigorous and fair comparisons of existing and future methods. In response to the reviewers’ suggestions, we have conducted substantial additional experiments and improved the manuscript. Most importantly, we have added sensitivity analyses to the main text (Sec. 4.4), which further confirm the robustness of our main conclusions. The revisions are summarized below.

  1. Figures (Reviewers TWHh and KXg4). We have redrawn Figure 1 and carefully improved the readability of the figures, which are now vector graphics wherever possible.
  2. Spatial partitions (Reviewer 3sPE). Following the reviewer's suggestion, we have further examined the effect of the spatial split under fixed 2 km blocks and a 1 km buffer, in addition to the finer 20×20 grid (Sec. 4.4, Table 3; App. E.2 and E.3). The top three models and the last position are unchanged under every partition, and random-split inflation appears under all of them.
  3. Alignment strategies (Reviewer KXg4). We have quantified the effect of different alignment strategies on all models (Sec. 4.4, Tables 3 and 4; App. E.3). The top five models retain their positions under both alignment groups, and the overall rankings remain highly consistent (Kendall's τ = 0.93), although some models in positions 6–10 change places. The main conclusions are robust to these tested alignment choices.
  4. Coverage (Reviewers 3sPE and KXg4). We have added the feature coverage of every model in every city, and re-evaluated all models only on the units covered by all 11 models (App. C.3). The ranking remains largely unchanged (τ = 0.91).
  5. Other revisions (Reviewer KXg4). We have added an analysis of label sources (App. E.4), jointly retrained the seven previously city-specific models for the cross-city transfer diagnostic (Sec. 4.5), and added the data-preparation pipelines and input metadata to the code repository.

We believe that the revised paper addresses the reviewers' main concerns, and we thank the reviewers again for their time and suggestions. We have uploaded the revised PDF with these changes.

We sincerely thank all reviewers for their time and constructive feedback. We have carefully addressed the concerns raised through additional experiments and revisions.

As the rebuttal period will end in three days, we would greatly appreciate any further feedback on our responses and revised manuscript. We are happy to provide additional clarification if needed.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.