- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
- This study proposes AgenticCityBench, a multi-modal, multi-temporal benchmark dataset for urban agents, which aims to connect urban physical environment and social activity data. The curated dataset covers 16 cities across 6 continents from 2000 to 2025, supporting two core directions: any-to-any cross-modal urban translation and agentic city tasks (prediction, planning, resource analysis). However, the tasks of this benchmark dataset are too broad and not sufficiently focused on concrete application domains.
- This study has made remarkable contributions in terms of data scale and multi-modal coverage, but has substantial weaknesses in data balance, temporal generalization, causal interpretation, joint training optimization, and evaluation methodology. It is suggested that the authors focus on solving the data imbalance problem, clearly distinguish the evaluation of static and dynamic modalities, deeply explore the mechanism of direction asymmetry, empirically validate the proposed joint training optimization strategy, and last but not least, narrow down the stake holder/application domain of this benchmark.
Critical strengths
-
The coverage and scale of AgenticCity-Bench are large: Coverage: 16 cities, 26 years (2000–2025), 18 semantic modalities, including built environment (building height and footprint, road network, land use/cover), activity data (population, electricity, carbon emissions, nighttime light, GPS density), environmental data (optical imagery, terrain, vegetation, weather, land surface temperature, air quality), etc. Data scale: At a resolution of 2km, there are a total of 7,488 city-year-modality combinations, among which 6,309 are available, with an overall availability rate of 84.3%.
-
This benchmark proposes translation in any direction between 18 Earth observation modalities, rather than being limited to specific source-target pairs. This design can systematically reveal the cross-modal complementarity and translation relationships.
Critical weaknesses
-
Imbalance in data availability, which may introduce systematic biases. There are significant differences in the modal coverage of this dataset (Table 10). For example, optical imagery, terrain, etc. achieve 100% availability, but the coverage rates of e.g. electricity (only 26.683%), real estate economy (55.769%), etc. are obviously insufficient. This means that the model may over-rely on high-coverage modalities during training, and the learning of low-coverage modalities can be insufficient. The evaluation results may not truly reflect the generalization ability of the model in data-scarce scenarios. In cross-modal translation tasks, the benchmark results of low-coverage modalities may be unreliable, affecting the universality of conclusions. These issues can be experimentally tested and discussed.
-
Reuse of static and epoch data may overestimate the temporal generalization of the model. The document clearly states that epoch-based data (such as built-up volume, population) and static data (such as terrain, building height) will be reused across multiple calendar years, and "do not constitute independent annual observations". This means that the predictability of the model in the temporal dimension may be overestimated, because it is actually learning repeated patterns rather than real temporal evolution. For modalities that rely on temporal changes (such as population, built-up volume), the model may fail to capture real annual dynamics. The evaluation metrics may not be able to distinguish whether the model truly understands temporal trends or merely memorizes static features. To account for this issue, the effect of the epoch-based data on the generalization of models can possibly be evaluated using simulation data (with continuous temporal variations) and comparing it to its sampling-to-epoch version. As this is a limitation of the data available, the authors need also acknowledge it to the reader.
-
The directional asymmetry of cross-modal translation lacks causal explanation. The document presents significant directional asymmetry in cross-modal translation. For example, DEM→WTH ranks second (NMSE 0.756%), while WTH→DEM ranks twelfth (NMAE 25.116%). Although the authors acknowledge that these relationships “should be interpreted as predictive associations rather than causal dependencies”, the paper does not explain the causes of this asymmetry. Therefore, the lack of theoretical explanation for the mechanism of asymmetry weakens the scientific contribution of the study. It fails to provide clear guidance for follow-up research, such as which modal combinations are complementary to each other under what conditions.
-
Evaluation metrics and normalization methods may mask the real differences between modalities. The document describes a variety of evaluation metrics (MSE, NMAE, mIoU, clDice, LPIPS, etc.), and applies the log1p transformation to targets with skewed distributions. Although this design takes the characteristics of different modalities into account, there are also potential problems: (1) the comparability between different metrics is questionable. For example, NMAE and mIoU measure completely different attributes, so direct comparison may be misleading; (2) percentile normalization ( and ) may be sensitive to outliers, affecting the stability of evaluation; (3) for multi-field numerical targets (such as weather, air quality), macro-averaging may mask the significant failures of some fields, making the overall performance appear better than the actual situation.
Questions for authors
Besides addressing the above-mentioned weaknesses, the following are some thoughts that the authors may consider to improve their paper:
-
Are there any specific application fields that the authors bear in mind? So that they can narrow down the applications of their proposed benchmark and using examples of concrete domains to organize their benchmark? To me, the translation tasks are for computing the basic environmental variables across geographic areas, it would make much sense to focus more on environmental or sustainable development issues rather than a one-bench-for-all ambition. Actually, urban or earth science are so complex and cannot possibly be fed into one benchmark.
-
How are urban dynamics reflected in the proposed benchmark? Currently, urban prediction and urban planning is related to temporal tasks, but these are far from urban dynamics. The authors can consider reduce their scope by changing the “urban dynamics” to more appropriate terms.
8 comments
W4:Evaluation metrics and normalization methods may mask the real differences between modalities.
A for W4: Thank you for highlighting these concerns. We have provided complementary evaluations for more insight between modalities.
(1) Comparability Across Target-Specific Metrics
We agree that NMAE, mIoU, clDice, and NMSE measure different attributes and should not be directly compared across target modalities. Our comparison is within each target modality: given a fixed target, we compare and rank alternative input modalities using the same target-specific metric and evaluation protocol.
(2) Percentile Normalization and Extreme Values
Urban values have lots of long-tail cases and pose a genuine modeling challenge. We acknowledge that percentile normalization alone does not eliminate sensitivity to such values. We therefore reported additional evaluation metrics in Appendix C.2 (Table 16) to provide complementary perspectives on prediction errors and reduce reliance on any single metric.
(3) Field-Wise Evaluation of Weather
We have added 6-field RMSE for weather in the revised manuscript (Appendix C.2 Table 17). Accurate temperature prediction doesn't mean reliable generation of the complete weather modality. PRCP score may be obscure.
Q1: Are there any specific application fields?
A for Q1: Yes, AgenticCity-Bench targets two application fields, urban science and remote sensing /earth observation. These applications align with the UN’s SDG 11, Sustainable Cities and Communities, and SDG 13, Climate Action [1].
(1) Any-to-any translation: For example, many urban records, such as electricity use or carbon emissions, are missing in data-scarce cities, while satellite imageries are available almost everywhere. Our any-to-any translation can be used to complete these missing modalities and to help urban researchers and policymakers understand relationships between different urban modalities.
(2) Agentic tasks: Our six agentic tasks correspond to decisions that governments face in practice.
For example, property price forecasting matters because housing is the largest household expense in most cities, and anticipating price changes supports affordability policy and land-use decisions (SDG 11). EV charging port planning also matters because charging availability is a main bottleneck for EV adoption now (SDG 7, SDG 11). The remaining tasks follow a similar logic. Each task requires combining urban function, human activity, and environmental observations over time, not a single data source, which is what our benchmark provides.
(3) Consistency with existing benchmarks: The applications of our benchmark are also consistent with recent urban benchmarks'. MapYourCity[2] evaluates building-age under geographic shifts and incomplete inputs. UrbanWell evaluates multimodal models’ ability to estimate urban wellbeing indicators, such as population and road length [3].
[1] United Nations. Transforming our world: the 2030 Agenda for Sustainable Development[R]. New York: United Nations, 2015. [2]Dionelis N, Feliciotti A, Marconcini M, et al. MapYourCity: A New Multimodal Benchmark Dataset and Community Challenge: Building age estimation from street-view and satellite imagery[J]. IEEE Geoscience and Remote Sensing Magazine, 2026. [3]Xi Y, Su X, Feng J, et al. UrbanWell: Benchmarking Multimodal Large Language Models for Spatio-Temporal Urban Wellbeing Analytics[C]//Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 2026: 10032-10043.
Q2: How are urban dynamics reflected in the benchmark?
A for Q2: Yes, Thank you for your comment. We will make this point clearer in the paper.
Following your suggestion, we have revised the paper accordingly in the abstract and main text of the paper. We have described agent tasks more explicitly as long-term and spatial reasoning tasks there.
Many developed cities change slowly. Prior work reports low growth rates and saturation in developed U.S. cities over 1993–2020 [1]. Such gradual change is only visible over long horizons. By covering 2000–2025, our benchmark spans enough time to capture it, and this long-term change is what we mean by urban dynamics. Recent remote-sensing benchmarks also examine urban change over long-term observations. DynamicVL uses multi-temporal imagery from 42 U.S. cities spanning 2005–2023 to evaluate dynamic city understanding [2]. Similarly, our benchmark collects complementary urban observations under a spatiotemporal index in a 25 years period.
[1] Hendrick M, Manoli G. Dynamic roughening of cities driven by multiplicative noise[J]. Nature Communications, 2026.
[2] Xuan W, Wang J, Qi H, et al. DynamicVL: Benchmarking multimodal large language models for dynamic city understanding[J]. Advances in Neural Information Processing Systems, 2026, 38.
Once again, we thank you for your time and insightful suggestions!
I appreciate the authors’ effort in their clarification in response to the questions raised, which has made the issues clearer to a certain extent. However, merely providing evidence from relevant studies may not necessarily fully address these questions. Different studies have their own applicable conditions, so the conclusions cannot always be generalized. In addition, some of the issues (such as the reuse of static data and epoch data, the impact of evaluation metrics and normalization methods, etc.) require further quantitative analysis to highlight the actual contribution/insights from this study, rather than being treated as supplementary discussions. As it stands, the contributions demonstrated by this study still do not sound very prominent to me in terms of its importance to the application field yet.
Dear Reviewer Zdu5,
We thank you for your careful and constructive review! Below we respond to each of your concern.
W1:Imbalance in data availability, which may introduce systematic biases.
A for W1: Yes, you are right. There is imbalance in data and such imbalance is a structural challenge in urban data collection[1]. Data-availability constraints are very common in urban and geospatial benchmarks. TraffiDent (NeurIPS 2025) reports missing traffic-volume data and excludes sensors with less than 50% observation coverage.MMEarth-Bench (ECCV 2026) documents missing Sentinel-1 SAR, canopy-height, precipitation, and temperature data.
There are also lots of reality constraints for data collection. In Tokyo, sharing individual smart-meter readings requires customer consent[2], and early electricity data lacks of public exposure. AFREC’s 2025 report highlights limited resources, costly surveys, and coordination challenges as barriers to energy-data collection in Africa [3].
This imbalance also provide directions for future research including data agmentation, cross-city transferring.
[1] Herfort B, Lautenbach S, Porto de Albuquerque J, et al. A spatio-temporal analysis investigating completeness and inequalities of global urban building data in OpenStreetMap[J]. Nature Communications, 2023, 14(1): 3985.
[2] Agency for Natural Resources and Energy, Ministry of Economy, Trade and Industry, Japan. (2025). Q{&}A on the Smart Meter Opt-Out System, Question 25. https://www.enecho.meti.go.jp/category/electricity_and_gas/electric/summary/regulations/smartmeter/index.html.
[3] African Energy Commission (AFREC). Africa Sustainable Development Goal 7 Report (2025 edition). African Union. Chapter 9, Sections 9.1–9.2, p. 78. https://au-afrec.org/sites/default/files/2026-01/Africa-SDG7.pdf
W2:Reuse of static and epoch data may overestimate the temporal generalization of the model.
A for W2: Thank you for your careful consideration. We collect lots of static data and epoch data in the dataset and align them with annual data. This is because different components of urban systems do not necessarily evolve at the same rate. Some built-up land and electricity infrastructure expand while population remains stable or lags behind[1]. It illustrates that population, built form, and infrastructure can follow different temporal trajectories.
For DEM, previous work [2] combines 2020 urban variables with SRTM elevation collected in the 2000s, explaining that this is appropriate given the relatively stable nature of broad-scale terrain. For population, five-year reference epochs are used by established products, including GHS-POP R2023A [3]. We retain these source values within each assigned interval rather than create interpolated annual estimates.
We will explain this more clearly in the revised manuscript.
[1] Li T, Wang Z, Kyba C, et al. Satellite imagery reveals increasing volatility in human night-time activity[J]. Nature, 2026, 652(8109): 379-386.
[2] Zhang Z, Qian Z, Chen M, et al. Worldwide rooftop photovoltaic electricity generation may mitigate global warming[J]. Nature Climate Change, 2025, 15(4): 393-402.
[3] Schiavina M, Melchiorri M, Pesaresi M, et al. GHSL Data Package 2023, Publications Office of the European Union[EB/OL].(2023)
W3:The directional asymmetry of cross-modal translation lacks causal explanation.
A for W3: Thanks for your careful review. We totally agree that there is directional asymmetry of cross-modal translation. The observed asymmetry can be understood in terms of the modalities’ spatial resolution, retained information, and physical relationships.
For example, DEM is informative for weather estimation while many different terrain configurations can be consistent with similar weather values. DEM preserves an elevation field at approximately 30-m resolution, whereas WTH derived from approximately 1-km weather products. Besides, DEM terrain provides physically relevant information about elevation-dependent temperature patterns and orographic precipitation [1]. So, DEM can support Weather generation effectively but Weathr provide limited information for DEM generation.
Recovering detailed spatial or visual outputs from compact numerical inputs is one of the capabilities explored by our any-to-any benchmark. This analysis suggests supplementing numerical summaries with spatially resolved inputs for detailed reconstruction.
[1]Chan W P, Lenoir J, Mai G S, et al. Climate velocities and species tracking in global mountain regions[J]. Nature, 2024, 629(8010): 114-120.