acceptodds
Under review as a conference paper at ICLR 2027

AgenticCity-Bench: Longitudinal Multimodal Observations of Urban Dynamics with Agentic Reasoning

20 views

Abstract

Urban systems are shaped by interactions among infrastructure, human activity, and environmental conditions. Existing earth observation and urban benchmarks are limited in unified modeling and in capturing their joint dynamics. We introduce AgenticCity-Bench, a longitudinal multimodal benchmark for multi-layer unified state modeling and agentic city task reasoning. The benchmark covers 16 cities in 12 countries across six continents from 2000 to 2025, and includes 18 modalities spanning urban function, activity, and environmental state. It organizes heterogeneous data sources around a common spatial reference and calendar index while preserving spatial and temporal coverage, provenance and tile-level availability. On this foundation, AgenticCity-Bench supports urban cross-modal translation together with diverse agentic city tasks involving forecasting, planning and resource analysis. This study provides a unified benchmark for studying the transition from multimodal urban and earth observation to joint understanding and agentic city decision-making. The dataset is available at https://anonymous.4open.science/r/AgenticCity-Bench.

Primary area:
datasets and benchmarks
Keywords:
Urban System, Earth Obversation, Agentic City, LLM.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Zdu5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  • This study proposes AgenticCityBench, a multi-modal, multi-temporal benchmark dataset for urban agents, which aims to connect urban physical environment and social activity data. The curated dataset covers 16 cities across 6 continents from 2000 to 2025, supporting two core directions: any-to-any cross-modal urban translation and agentic city tasks (prediction, planning, resource analysis). However, the tasks of this benchmark dataset are too broad and not sufficiently focused on concrete application domains.
  • This study has made remarkable contributions in terms of data scale and multi-modal coverage, but has substantial weaknesses in data balance, temporal generalization, causal interpretation, joint training optimization, and evaluation methodology. It is suggested that the authors focus on solving the data imbalance problem, clearly distinguish the evaluation of static and dynamic modalities, deeply explore the mechanism of direction asymmetry, empirically validate the proposed joint training optimization strategy, and last but not least, narrow down the stake holder/application domain of this benchmark.
Critical strengths
  • The coverage and scale of AgenticCity-Bench are large: Coverage: 16 cities, 26 years (2000–2025), 18 semantic modalities, including built environment (building height and footprint, road network, land use/cover), activity data (population, electricity, carbon emissions, nighttime light, GPS density), environmental data (optical imagery, terrain, vegetation, weather, land surface temperature, air quality), etc. Data scale: At a resolution of 2km, there are a total of 7,488 city-year-modality combinations, among which 6,309 are available, with an overall availability rate of 84.3%.

  • This benchmark proposes translation in any direction between 18 Earth observation modalities, rather than being limited to specific source-target pairs. This design can systematically reveal the cross-modal complementarity and translation relationships.

Critical weaknesses
  1. Imbalance in data availability, which may introduce systematic biases. There are significant differences in the modal coverage of this dataset (Table 10). For example, optical imagery, terrain, etc. achieve 100% availability, but the coverage rates of e.g. electricity (only 26.683%), real estate economy (55.769%), etc. are obviously insufficient. This means that the model may over-rely on high-coverage modalities during training, and the learning of low-coverage modalities can be insufficient. The evaluation results may not truly reflect the generalization ability of the model in data-scarce scenarios. In cross-modal translation tasks, the benchmark results of low-coverage modalities may be unreliable, affecting the universality of conclusions. These issues can be experimentally tested and discussed.

  2. Reuse of static and epoch data may overestimate the temporal generalization of the model. The document clearly states that epoch-based data (such as built-up volume, population) and static data (such as terrain, building height) will be reused across multiple calendar years, and "do not constitute independent annual observations". This means that the predictability of the model in the temporal dimension may be overestimated, because it is actually learning repeated patterns rather than real temporal evolution. For modalities that rely on temporal changes (such as population, built-up volume), the model may fail to capture real annual dynamics. The evaluation metrics may not be able to distinguish whether the model truly understands temporal trends or merely memorizes static features. To account for this issue, the effect of the epoch-based data on the generalization of models can possibly be evaluated using simulation data (with continuous temporal variations) and comparing it to its sampling-to-epoch version. As this is a limitation of the data available, the authors need also acknowledge it to the reader.

  3. The directional asymmetry of cross-modal translation lacks causal explanation. The document presents significant directional asymmetry in cross-modal translation. For example, DEM→WTH ranks second (NMSE 0.756%), while WTH→DEM ranks twelfth (NMAE 25.116%). Although the authors acknowledge that these relationships “should be interpreted as predictive associations rather than causal dependencies”, the paper does not explain the causes of this asymmetry. Therefore, the lack of theoretical explanation for the mechanism of asymmetry weakens the scientific contribution of the study. It fails to provide clear guidance for follow-up research, such as which modal combinations are complementary to each other under what conditions.

  4. Evaluation metrics and normalization methods may mask the real differences between modalities. The document describes a variety of evaluation metrics (MSE, NMAE, mIoU, clDice, LPIPS, etc.), and applies the log1p transformation to targets with skewed distributions. Although this design takes the characteristics of different modalities into account, there are also potential problems: (1) the comparability between different metrics is questionable. For example, NMAE and mIoU measure completely different attributes, so direct comparison may be misleading; (2) percentile normalization (p01p_{01} and p99p_{99}) may be sensitive to outliers, affecting the stability of evaluation; (3) for multi-field numerical targets (such as weather, air quality), macro-averaging may mask the significant failures of some fields, making the overall performance appear better than the actual situation.

Questions for authors

Besides addressing the above-mentioned weaknesses, the following are some thoughts that the authors may consider to improve their paper:

  • Are there any specific application fields that the authors bear in mind? So that they can narrow down the applications of their proposed benchmark and using examples of concrete domains to organize their benchmark? To me, the translation tasks are for computing the basic environmental variables across geographic areas, it would make much sense to focus more on environmental or sustainable development issues rather than a one-bench-for-all ambition. Actually, urban or earth science are so complex and cannot possibly be fed into one benchmark.

  • How are urban dynamics reflected in the proposed benchmark? Currently, urban prediction and urban planning is related to temporal tasks, but these are far from urban dynamics. The authors can consider reduce their scope by changing the “urban dynamics” to more appropriate terms.

8 comments

Dear Reviewer Zdu5,

We thank you for your careful and constructive review! Below we respond to each of your concern.

W1:Imbalance in data availability, which may introduce systematic biases.

A for W1: Yes, you are right. There is imbalance in data and such imbalance is a structural challenge in urban data collection[1]. Data-availability constraints are very common in urban and geospatial benchmarks. TraffiDent (NeurIPS 2025) reports missing traffic-volume data and excludes sensors with less than 50% observation coverage.MMEarth-Bench (ECCV 2026) documents missing Sentinel-1 SAR, canopy-height, precipitation, and temperature data.

There are also lots of reality constraints for data collection. In Tokyo, sharing individual smart-meter readings requires customer consent[2], and early electricity data lacks of public exposure. AFREC’s 2025 report highlights limited resources, costly surveys, and coordination challenges as barriers to energy-data collection in Africa [3].

This imbalance also provide directions for future research including data agmentation, cross-city transferring.

[1] Herfort B, Lautenbach S, Porto de Albuquerque J, et al. A spatio-temporal analysis investigating completeness and inequalities of global urban building data in OpenStreetMap[J]. Nature Communications, 2023, 14(1): 3985.

[2] Agency for Natural Resources and Energy, Ministry of Economy, Trade and Industry, Japan. (2025). Q{&}A on the Smart Meter Opt-Out System, Question 25. https://www.enecho.meti.go.jp/category/electricity_and_gas/electric/summary/regulations/smartmeter/index.html.

[3] African Energy Commission (AFREC). Africa Sustainable Development Goal 7 Report (2025 edition). African Union. Chapter 9, Sections 9.1–9.2, p. 78. https://au-afrec.org/sites/default/files/2026-01/Africa-SDG7.pdf

W2:Reuse of static and epoch data may overestimate the temporal generalization of the model.

A for W2: Thank you for your careful consideration. We collect lots of static data and epoch data in the dataset and align them with annual data. This is because different components of urban systems do not necessarily evolve at the same rate. Some built-up land and electricity infrastructure expand while population remains stable or lags behind[1]. It illustrates that population, built form, and infrastructure can follow different temporal trajectories.

For DEM, previous work [2] combines 2020 urban variables with SRTM elevation collected in the 2000s, explaining that this is appropriate given the relatively stable nature of broad-scale terrain. For population, five-year reference epochs are used by established products, including GHS-POP R2023A [3]. We retain these source values within each assigned interval rather than create interpolated annual estimates.

We will explain this more clearly in the revised manuscript.

[1] Li T, Wang Z, Kyba C, et al. Satellite imagery reveals increasing volatility in human night-time activity[J]. Nature, 2026, 652(8109): 379-386.

[2] Zhang Z, Qian Z, Chen M, et al. Worldwide rooftop photovoltaic electricity generation may mitigate global warming[J]. Nature Climate Change, 2025, 15(4): 393-402.

[3] Schiavina M, Melchiorri M, Pesaresi M, et al. GHSL Data Package 2023, Publications Office of the European Union[EB/OL].(2023)

W3:The directional asymmetry of cross-modal translation lacks causal explanation.

A for W3: Thanks for your careful review. We totally agree that there is directional asymmetry of cross-modal translation. The observed asymmetry can be understood in terms of the modalities’ spatial resolution, retained information, and physical relationships.

For example, DEM is informative for weather estimation while many different terrain configurations can be consistent with similar weather values. DEM preserves an elevation field at approximately 30-m resolution, whereas WTH derived from approximately 1-km weather products. Besides, DEM terrain provides physically relevant information about elevation-dependent temperature patterns and orographic precipitation [1]. So, DEM can support Weather generation effectively but Weathr provide limited information for DEM generation.

Recovering detailed spatial or visual outputs from compact numerical inputs is one of the capabilities explored by our any-to-any benchmark. This analysis suggests supplementing numerical summaries with spatially resolved inputs for detailed reconstruction.

[1]Chan W P, Lenoir J, Mai G S, et al. Climate velocities and species tracking in global mountain regions[J]. Nature, 2024, 629(8010): 114-120.

W4:Evaluation metrics and normalization methods may mask the real differences between modalities.

A for W4: Thank you for highlighting these concerns. We have provided complementary evaluations for more insight between modalities.

(1) Comparability Across Target-Specific Metrics

We agree that NMAE, mIoU, clDice, and NMSE measure different attributes and should not be directly compared across target modalities. Our comparison is within each target modality: given a fixed target, we compare and rank alternative input modalities using the same target-specific metric and evaluation protocol.

(2) Percentile Normalization and Extreme Values

Urban values have lots of long-tail cases and pose a genuine modeling challenge. We acknowledge that percentile normalization alone does not eliminate sensitivity to such values. We therefore reported additional evaluation metrics in Appendix C.2 (Table 16) to provide complementary perspectives on prediction errors and reduce reliance on any single metric.

(3) Field-Wise Evaluation of Weather

We have added 6-field RMSE for weather in the revised manuscript (Appendix C.2 Table 17). Accurate temperature prediction doesn't mean reliable generation of the complete weather modality. PRCP score may be obscure.

Q1: Are there any specific application fields?

A for Q1: Yes, AgenticCity-Bench targets two application fields, urban science and remote sensing /earth observation. These applications align with the UN’s SDG 11, Sustainable Cities and Communities, and SDG 13, Climate Action [1].

(1) Any-to-any translation: For example, many urban records, such as electricity use or carbon emissions, are missing in data-scarce cities, while satellite imageries are available almost everywhere. Our any-to-any translation can be used to complete these missing modalities and to help urban researchers and policymakers understand relationships between different urban modalities.

(2) Agentic tasks: Our six agentic tasks correspond to decisions that governments face in practice.

For example, property price forecasting matters because housing is the largest household expense in most cities, and anticipating price changes supports affordability policy and land-use decisions (SDG 11). EV charging port planning also matters because charging availability is a main bottleneck for EV adoption now (SDG 7, SDG 11). The remaining tasks follow a similar logic. Each task requires combining urban function, human activity, and environmental observations over time, not a single data source, which is what our benchmark provides.

(3) Consistency with existing benchmarks: The applications of our benchmark are also consistent with recent urban benchmarks'. MapYourCity[2] evaluates building-age under geographic shifts and incomplete inputs. UrbanWell evaluates multimodal models’ ability to estimate urban wellbeing indicators, such as population and road length [3].

[1] United Nations. Transforming our world: the 2030 Agenda for Sustainable Development[R]. New York: United Nations, 2015. [2]Dionelis N, Feliciotti A, Marconcini M, et al. MapYourCity: A New Multimodal Benchmark Dataset and Community Challenge: Building age estimation from street-view and satellite imagery[J]. IEEE Geoscience and Remote Sensing Magazine, 2026. [3]Xi Y, Su X, Feng J, et al. UrbanWell: Benchmarking Multimodal Large Language Models for Spatio-Temporal Urban Wellbeing Analytics[C]//Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 2026: 10032-10043.

Q2: How are urban dynamics reflected in the benchmark?

A for Q2: Yes, Thank you for your comment. We will make this point clearer in the paper.

Following your suggestion, we have revised the paper accordingly in the abstract and main text of the paper. We have described agent tasks more explicitly as long-term and spatial reasoning tasks there.

Many developed cities change slowly. Prior work reports low growth rates and saturation in developed U.S. cities over 1993–2020 [1]. Such gradual change is only visible over long horizons. By covering 2000–2025, our benchmark spans enough time to capture it, and this long-term change is what we mean by urban dynamics. Recent remote-sensing benchmarks also examine urban change over long-term observations. DynamicVL uses multi-temporal imagery from 42 U.S. cities spanning 2005–2023 to evaluate dynamic city understanding [2]. Similarly, our benchmark collects complementary urban observations under a spatiotemporal index in a 25 years period.

[1] Hendrick M, Manoli G. Dynamic roughening of cities driven by multiplicative noise[J]. Nature Communications, 2026.

[2] Xuan W, Wang J, Qi H, et al. DynamicVL: Benchmarking multimodal large language models for dynamic city understanding[J]. Advances in Neural Information Processing Systems, 2026, 38.

Once again, we thank you for your time and insightful suggestions!

I appreciate the authors’ effort in their clarification in response to the questions raised, which has made the issues clearer to a certain extent. However, merely providing evidence from relevant studies may not necessarily fully address these questions. Different studies have their own applicable conditions, so the conclusions cannot always be generalized. In addition, some of the issues (such as the reuse of static data and epoch data, the impact of evaluation metrics and normalization methods, etc.) require further quantitative analysis to highlight the actual contribution/insights from this study, rather than being treated as supplementary discussions. As it stands, the contributions demonstrated by this study still do not sound very prominent to me in terms of its importance to the application field yet.

Reviewer_kUaUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AgenticCity-Bench, a longitudinal urban benchmark integrating 18 modalities across 16 cities over 2000–2025. It organizes heterogeneous built-environment, human-activity, and environmental data into shared spatial grids, with metadata describing data sources, temporal coverage, and missingness.

The benchmark supports two evaluation tracks: (1) any-to-any cross-modal translation across 306 directed modality pairs, using fine-tuned multimodal generative models; and (2) 1,000 agentic cases across six tasks involving urban forecasting, EV charging expansion, school capacity planning, and food accessibility. Experiments compare general-purpose and domain-specific agents against historical baselines, revealing uneven cross-modal generation performance and limitations in agents' ability to translate observations into quantitative decisions.

Critical strengths
  1. The benchmark integrates 18 heterogeneous urban modalities across 16 cities and 26 years, providing a useful resource for studying urban dynamics and multimodal learning.

  2. The paper introduces cross-modal translation and six agentic tasks covering forecasting, infrastructure planning, and resource analysis, connecting multimodal modeling with practical urban problems.

  3. The experiments reveal asymmetric cross-modal translation performance, potential interference between modalities, and limitations in agents' ability to translate urban observations into accurate quantitative predictions and decisions.

Critical weaknesses
  1. The reported 84.3% availability measures city–year–modality presence rather than tile-level coverage. Static and five-year observations are also reused across years.

  2. Translation only evaluates one-to-one mappings, without testing multimodal complementarity or isolating the contribution of source modalities.

  3. All four generative models are fine-tuned, without zero-shot or task-specific baselines. Cross-model comparisons cover only five translation routes.

Questions for authors
  1. Table 9 lists land-cover observations through 2022, whereas Table 10 reports full availability through 2025. How is this discrepancy explained?

  2. What is the actual tile-level and joint-modality coverage across cities and years?

3 comments

Dear Reviewer kUaU,

We thank you for your careful and constructive review. Below we respond to each of your concerns.

W1: The reported 84.3% availability measures city–year–modality presence rather than tile-level coverage. Static and five-year observations are also reused across years.

Q2: What is the actual tile-level and joint-modality coverage across cities and years?

A for W1/Q2: Thank you for identifying this. You are right, the reported 84.3% is a city–year–modality presence statistic, not tile-level coverage. The collection contains annual, epoch-based, and static records. Urban variables do not evolve at one common rate; slowly varying background geography coexists with rapidly changing weather, activity, or construction[1].

For Dem, previous work[2] combines 2020 urban variables with elevation acquired in the 2000s. Older elevation layer remains appropriate because broad-scale terrain is relatively stable. For population, GHS-POP R2023A [3] provides population estimates at five-year intervals rather than separate estimates for every year. We reuse the corresponding population layer within each assigned five-year interval following previous work.

Our temporal rule is explicit: annual records use their reported year; five-year products are reused only within their assigned interval; and static products retain the same source layer across indexed years. We will clarify in the documentation when each source was collected, which year or period it represents, and whether it is reused across years.

In the abstract link, we prepare a year-by-modality heatmap. Each cell reports the number of distinct admitted tiles for that city, year, and modality, divided by the city’s total number of grid tiles.

The following table show the overall city-level availability by modality and will be added to paper.

CityTilesBVRDLCLUHFELECNTLCO2GPSPOPPROPOPTDEMNDVIWTHHEATAQOverall
Amsterdam140100.073.174.780.8100.034.053.896.292.3100.01.4100.0100.096.396.249.688.578.1
Boston169100.073.1100.056.4100.023.9100.096.269.2100.032.5100.0100.099.994.150.088.580.7
Busan208100.073.1100.050.9100.033.6100.096.265.491.829.099.8100.099.496.250.088.580.2
Cape Town182100.073.1100.072.8100.021.9100.096.269.2100.011.1100.0100.098.396.250.088.580.4
Christchurch960100.073.1100.043.20.014.5100.096.273.182.60.097.9100.098.996.250.088.571.3
Washington, D.C.143100.069.2100.074.6100.032.553.895.565.4100.042.7100.0100.099.5100.050.088.580.0
London720100.073.163.680.8100.042.2100.096.292.3100.096.3100.0100.098.296.250.088.585.9
Lyon143100.073.186.480.8100.041.8100.096.269.2100.018.1100.0100.0100.096.250.088.581.6
New York City624100.073.1100.044.4100.024.1100.096.269.289.944.499.7100.099.580.050.088.579.3
San Francisco323100.073.1100.034.3100.016.853.896.269.291.015.0100.0100.099.096.250.088.575.1
Santiago182100.073.1100.080.8100.026.9100.096.269.2100.00.0100.0100.099.696.250.088.580.5
Shanghai169100.069.2100.080.80.026.953.896.219.2100.027.5100.0100.099.996.250.088.571.0
Singapore640100.073.1100.024.0100.019.4100.096.261.581.214.399.3100.083.296.250.088.575.3
Sydney169100.073.1100.073.1100.023.4100.096.276.9100.099.9100.0100.097.296.250.088.585.8
Tokyo289100.073.1100.061.5100.020.153.896.276.994.844.2100.0100.097.496.250.088.579.0
Xiong’an775100.065.4100.080.8100.023.1100.096.253.8100.00.0100.0100.0100.096.250.088.579.1

[1] Li T, Wang Z, Kyba C, et al. Satellite imagery reveals increasing volatility in human night-time activity[J]. Nature, 2026, 652(8109): 379-386.

[2] Zhang Z, Qian Z, Chen M, et al. Worldwide rooftop photovoltaic electricity generation may mitigate global warming[J]. Nature Climate Change, 2025, 15(4): 393-402.

[3] Schiavina M, Melchiorri M, Pesaresi M, et al. GHSL Data Package 2023, Publications Office of the European Union[EB/OL].(2023)

W2: Translation only evaluates one-to-one mappings, without testing multimodal complementarity or isolating the contribution of source modalities.

A for W2: Thank you for highlighting this point. We have added two-to-one and three-to-one translation experiments to evaluate multimodal complementarity beyond the original one-to-one setting. These results provide evidence of multimodal complementarity. The three-source combination achieving the lowest error .

(1)Joint-training results below

(a) WTH target

InputsTargetmacro NMSE (%) ↓
NDVIWTH1.177518
DEMWTH5.277934
LCWTH5.442
NDVI + DEMWTH0.734
NDVI + LCWTH1.165
DEM + LCWTH0.751
NDVI + DEM + LCWTH0.161

(b) POP target

InputsTargetNMSE (%) ↓
H&FPOP10.124191
LUPOP9.808
NTLPOP4.084
H&F + LUPOP9.607
H&F + NTLPOP3.497
LU + NTLPOP4.389
H&F + LU + NTLPOP4.002

(c) OPT target

InputsTargetMSE ↓
WTHOPT0.0367
LCOPT0.0299
H&FOPT0.0304
WTH + LCOPT0.0297
WTH + H&FOPT0.0302
LC + H&FOPT0.0296
WTH + LC + H&FOPT0.0251

(2) Only any-to-any training below

(a) WTH Target

InputsTargetmacro NMSE (%) ↓
NDVIWTH12.397
DEMWTH0.756
LCWTH1.442
NDVI + DEMWTH0.392
NDVI + LCWTH0.388
DEM + LCWTH0.370
NDVI + DEM + LCWTH0.363

(b) POP Target

InputsTargetNMSE (%) ↓
H&FPOP20.229
LUPOP28.571
NTLPOP4.400
H&F + LUPOP9.131
H&F + NTLPOP2.382
LU + NTLPOP2.561
H&F + LU + NTLPOP1.747

(c) OPT Target

InputsTargetMSE ↓
WTHOPT0.126
LCOPT0.113
H&FOPT0.120
WTH + LCOPT0.0559
WTH + H&FOPT0.0542
LC + H&FOPT0.0544
WTH + LC + H&FOPT0.0515

W3:All four generative models are fine-tuned, without zero-shot or task-specific baselines. Cross-model comparisons cover only five translation routes.

A for W3: Thank you for this comment. Follow your suggestion, we have evaluated available pretrained checkpoints in a zero-shot setting, with results below.

ModelNDVI→WTHPOI→ELECH&F→POPWTH→OPTOPT→RD
Metricmacro NMSE (%)↓log1p NMSE (%)↓log1p NMSE (%)↓RGB MSE↓clDice↑
4M-21-B (0.834B)INVINVINV0.66420.0058
Show-o2 (7B)INVINVINV0.41290.0060
UniGen (1.5B)-----
MODUS (7B)INVINVINV0.22940.0000

INV indicates that the model generates invalid outputs. UniGen is marked - because no official pretrained checkpoint was available.

The following table show the examples of zero-shot invalid outputs.

ModelTaskRaw output (excerpt)Invalid-output reason
4M-21-BNDVI + context; predict weathercity = amsterdam ; … ; answer =.Echoes the prompt without generating an answer.
MODUSNDVI + context; predict weatherThe image appears to be a ….Produces a caption rather than the required numerical JSON.
Show-o2NDVI + context; predict weather"ndvi":0.5Returns valid JSON for the wrong modality.

Without adaptation to our structured urban numerical modalities, the official pretrained checkpoints failed to generate valid numerical outputs. These results indicate that our benchmark is necessary for urban translation tasks and motivates benchmark-specific adaptation.

Besides, we also supplement the more cross-model results in Appendix C.2 Tables 18-20.

Q1: Table 9 lists land-cover observations through 2022, whereas Table 10 reports full availability through 2025. How is this discrepancy explained?

A for Q1: Sorry for the inconsistency between Tables 9 and 10. The cited LC-FCS30D [1] product genuinely ends in 2022; “2022” is therefore not the incorrect source endpoint. We supplement the collection with annual land-cover composites derived from Dynamic World [2]. Its continuously updated archive includes data for 2023–2025. So the land cover observation has full availability through 2025. We will check relative descriptions and correct it in revised manuscript.

[1] Zhang X, Zhao T, Xu H, et al. GLC-FCS30D: the first global 30 m land-cover dynamics monitoring product with a fine classification system for the period from 1985 to 2022 generated using dense-time-series Landsat imagery and the continuous change-detection method[J]. Earth System Science Data, 2024, 16(3): 1353-1381.

[2] Brown C F, Brumby S P, Guzder-Williams B, et al. Dynamic World, Near real-time global 10 m land use land cover mapping[J]. Scientific data, 2022, 9(1): 251.

Once again, we thank you for your time and insightful suggestions!

Thank you for your rebuttal. I will maintain my positive score.

Reviewer_E5VnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents AgenticCity-Bench, a longitudinal multimodal urban dataset covering 16 cities in 12 countries from 2000 to 2025. It contains 18 modalities grouped into urban function, activity, and environment, aligned on 2 km and 5 km tiles. Two benchmark tracks are built on it. The first is any-to-any translation between modalities, for which four unified multimodal models are fine-tuned. The second consists of 1,000 agentic cases covering forecasting, planning, and food-access analysis, evaluated with general coding agents and a domain-specific Earth agent.

Critical strengths
  • Aligning heterogeneous physical and socioeconomic urban data across 16 cities and 26 years is a substantial data effort.
  • The data sources and availability masks are documented.
Critical weaknesses
  • It is unclear whether the tasks are hard enough to require agents. For LST, the ten-year mean outperforms all agents, and for food access, cell accuracy is close to the majority baseline.
  • The "planning" tasks are essentially classification. EV planning predicts one of four growth ranges, and food access assigns one of three labels per cell.
  • Trajectories are not evaluated. Only final outputs and tool-call counts are reported, and the validity of tool selection, call order, and arguments is not assessed.
  • The geographic coverage is biased. The agentic tasks use only four US cities and London, and Africa and South America are each represented by a single city.
  • The quality of the task labels is not verified. Several targets are model-derived or spatially mismatched.
  • In the translation track, no context-only baseline is reported, although the prompt always contains the city, year, and coordinates. A comparison with TerraMind (ICCV 2025) is also missing.
Questions for authors
  • How long are the agent trajectories (number of steps and tokens), and how much do results vary across repeated runs?
  • How does a translation model that receives only the grid context (no source modality) perform?
  • How were the quality and consistency of the task labels verified beyond the data availability checks?
5 comments

Dear Reviewer E5Vn,

Thank you for your detailed and constructive review! Below we respond to each of your concerns.

Q1: How long are the agent trajectories (number of steps and tokens), and how much do results vary across repeated runs?

A for Q1: Thank you for your suggestion. We have added detailed token usage and step results by task to the revised manuscript in Appendix D.4.1. The table below summarizes agent token and step usage from the original evaluation. Appendix D.4.1 (Tables 43–45) also provides tool-use statistics and error rates by task.

Agent configurationCasesStepsTokens (k)
Earth Agent (GPT Luna)6005.15 ± 1.00145.0 ± 41.1
Earth Agent (GPT Terra)4004.67 ± 1.8457.3 ± 42.3
Earth Agent (DeepSeek)1,0008.70 ± 2.58333.2 ± 209.2
Codex (GPT-5.5)1,0007.30 ± 2.87163.1 ± 117.8
OpenCode (GPT Luna)60012.65 ± 1.81270.1 ± 66.4
OpenCode (GPT Terra)4004.77 ± 1.5954.8 ± 34.5
OpenCode (Gemini)1,0009.09 ± 4.41234.5 ± 189.9

For repeated-run variability, for example, we randomly selected five property-price forecasting cases and performed three independent runs per case, keeping each agent’s configuration fixed across repetitions. The results suggest reasonably consistent performance on this five-case subset. Upon acceptance, we will release the full benchmark tasks, agent configurations, and detailed trajectory logs to support replication of our experiments.

Agent configurationRun 1 MAPE (%)Run 2Run 3Mean ± SD
Earth Agent (GPT Luna)9.358.997.928.75 ± 0.75
Earth Agent (DeepSeek)7.006.726.436.72 ± 0.29
OpenCode (GPT Luna)6.274.788.386.48 ± 1.80
OpenCode (Gemini)5.597.956.856.80 ± 1.18

Q2: How does a translation model that receives only the grid context (no source modality) perform?

W6: In the translation track, no context-only baseline is reported, although the prompt always contains the city, year, and coordinates. A comparison with TerraMind (ICCV 2025) is also missing.

A for W6/Q2: Thank you for your careful review. TerraMind and the context-only comparison are essential to the evaluation.

TerraMind doesn't provide a official repo with numerical-input interface. We implemented the text interface following the design in the TerraMind paper and extended it to support our numerical modalities.

ModelsNDVI→WTHPOI→ELECHF→POPWTH→OPTOPT→RD
Metricmacro NMSE (%)↓NMSE (%)↓NMSE (%)↓MSE↓clDice↑
ML-4M-3B77.80032.300106.1000.0630.139
Show-O2-1.5B53.5003.324164.2000.2990.024
UniGen-1.5B19.00096.90087.9000.036—
MODUS-7B12.39737.50020.2290.1260.288
TerraMind-1B76.4864.00529.9180.0720.008

These results still support our finding that model strengths vary across source–target pairs, with no single model dominating all evaluated routes.

We additionally evaluated a context-only MODUS baseline that receives grid context without any source-modality inputs.

Model / input settingPOI→ELECHF→POPWTH→OPTOPT→RD
MetricNMSE (%)↓NMSE (%)↓MSE↓clDice↑
MODUS-7B37.50020.2290.1260.288
MODUS-7B context-only46.80438.4740.1340.244

The standard model performs better on all test sets and the advantage is uniform across targets.

Q3: How were the quality and consistency of the task labels verified beyond the data availability checks?

W5: The quality of the task labels is not verified. Several targets are model-derived or spatially mismatched.

A for Q3/W5: Thank you for your concern. We didn't make this point clear. Our quality control focuses on verifying spatial and temporal consistency across modalities, while the accuracy and uncertainty of the original measurements are documented by their respective data providers.

Within AgenticCity-Bench, each sample is indexed by (city, scale, row, column, year), where the row and column identify a fixed projected 2 × 2 km or 5 × 5 km tile. Spatial alignment is checked using coordinate reference systems and tile boundaries. Temporal alignment is checked against acquisition dates, reporting periods, and permitted reuse intervals for static or epoch-based products. Following automated alignment, we manually inspected 50 randomly selected samples from each city and found no spatial or temporal alignment inconsistencies among the available modalities in the inspected samples.

For source-product quality, the source websites are listed in Appendix B.3, Table 8: Primary source registry for the AgenticCity-Bench modalities. Supporting publications, including those for ODIAC [1] and VIIRS nighttime lights [2], document source-product construction and quality assessment.

[1] Oda T, Maksyutov S, Andres R J. The Open-source Data Inventory for Anthropogenic CO 2, version 2016 (ODIAC2016): a global monthly fossil fuel CO 2 gridded emissions data product for tracer transport simulations and surface flux inversions[J]. Earth System Science Data, 2018, 10(1): 87-107.

[2] Elvidge C D, Zhizhin M, Ghosh T, et al. Annual time series of global VIIRS nighttime lights derived from monthly averages: 2012 to 2019[J]. Remote Sensing, 2021, 13(5): 922.

W1: It is unclear whether the tasks are hard enough to require agents. For LST, the ten-year mean outperforms all agents, and for food access, cell accuracy is close to the majority baseline.

A for W1:

We apologize for the confusion. We agree that benchmarks should set complex tasks to evaluate the full range of agent abilities.

Forecasting tasks are widely studied in urban-agent and urban-LLM research, including CityBench [1], USTBench [2], UrbanGPT [3], AgentMove [4], and UrbanLLM [5]. An inaccurate prediction of enrollment can lead to a poor capacity decision, as shown in our school capacity planning task, and prediction errors can accumulate in multi-step planning.

AgenticCity-Bench aims to provide a longitudinal dataset and fair task settings to evaluate agents’ understanding of cities and to identify the limitations of current urban agents. LST is a long-term indicator that requires agents to capture the long-term trend in a physical modality. Performance on the LST task reveals that current agents’ understanding of urban-specific modalities is insufficient. For food access, cell accuracy is close to the majority baseline because the labels are imbalanced. But on macro-F1, which accounts for this, the best agent reaches 0.475 against 0.221 for the baseline. In the remaining two forecasting tasks, agents also need to make full use of relevant modalities, carbon emissions and property prices. On carbon emissions, all agents outperform the historical baselines, and on property prices the best agents do.

[1] Feng J, Zhang J, Liu T, et al. CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks[C]//Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2025.

[2] Lai S, Ning Y, Yuan Z, et al. USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agents[EB/OL]. arXiv:2505.17572, 2025.

[3] Li Z, Xia L, Tang J, et al. UrbanGPT: Spatio-Temporal Large Language Models[C]//Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024: 5351–5362.

[4] Feng J, Du Y, Zhao J, et al. AgentMove: A Large Language Model based Agentic Framework for Zero-shot Next Location Prediction[C]//Proceedings of NAACL: Human Language Technologies (Volume 1: Long Papers). 2025: 1322–1338.

[5] Jiang Y, Chao Q, Chen Y, et al. Urbanllm: Autonomous urban activity planning and management with large language models[C]//Findings of the Association for Computational Linguistics: EMNLP 2024. 2024: 1810-1825.

W2: The "planning" tasks are essentially classification. EV planning predicts one of four growth ranges, and food access assigns one of three labels per cell.

A for W2: Thank you for the comment. We should have made this clearer. Actually, four of our six tasks (700 out of 1,000 cases) require numerical outputs and all tasks need to provide supporting reasons. The school capacity planning task requires agents to specify the exact number of new student places, considering both future enrollment demand and existing school capacity.

For EV expansion planning, the four options represent ranges of net planned charging-port construction over two years. Agents must consider how factors such as population, charging demand, and electricity use may change over this horizon, which goes beyond simple classification. For food-access analysis, the output is a label, but it depends on two quantities (household income and distance to a supermarket) that are not given to the agent. Agents must infer both from indirect multimodal evidence (e.g., property values, POI, population) for each of four cells and cite the evidence they used.

W3: Trajectories are not evaluated. Only final outputs and tool-call counts are reported, and the validity of tool selection, call order, and arguments is not assessed.

A for W3: Thank you for your suggestion. To supplement, we further analyzed agents’ access rates for urban modalities in the revised manuscript. Appendix Figure 6 summarizes modality access by Codex (GPT-5.5) across 600 forecasting cases.

We provided analyses of agents’ reasoning and evidence use in Appendix D.3.2, alongside tool-use statistics and execution-error rates in Appendix D.4.1 as answered in Q1. Because our benchmark requires agents to plan their own tool use, tool selection and call order are difficult to rate. Our focus is whether agents can cope with many city modalities and their relationships.

W4: The geographic coverage is biased. The agentic tasks use only four US cities and London, and Africa and South America are each represented by a single city.

A for W4: Thank you for raising this concern. Geographic representation was an explicit consideration in our task design.

Africa and South America are represented by Cape Town and Santiago, respectively. Expanding this coverage requires reliable, spatially aligned, longitudinal data at tile resolution across the relevant modalities. This challenge sits within the broader urban-data availability gaps documented by UN-Habitat [1].

The prediction tasks cover 16 cities across 12 countries, with 59.17% outside the US and UK. The table below provides the forecasting case distribution. Our paper also provides detailed city-level performance and uncertainty estimates in Appendix Tables 25–28 and Appendix Tables 29–31. These results document geographic variation in agent performance alongside the benchmark’s coverage.

Country / US cityLSTCarbonPropertyTotalShare(%)
Boston151020457.50
New York City151520508.33
San Francisco151520508.33
Washington, DC151520508.33
South Korea151520508.33
United Kingdom151520508.33
Australia151020457.50
Singapore101520457.50
China15250406.67
Chile15150305.00
France15150305.00
Netherlands15150305.00
New Zealand15150305.00
South Africa15150305.00
Japan15100254.17
Total220220160600100

[1] UN-Habitat. Transforming Global Urban Data 2023–2029[EB/OL].

Once again, we thank you for your time and insightful suggestions!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Looking Forward to Further Discussions!

Dear Reviewer,

We were wondering if our responses have cleared all your concerns. We have carefully considered all the points you raised and have provided corresponding clarifications and additional results in our rebuttal.

We would greatly appreciate your valuable feedback and would be happy to discuss it further. We hope our discussion can help improve the clarity and quality of our manuscript.

Thank you very much for your time and consideration!

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.