acceptodds
Under review as a conference paper at ICLR 2027

DUDA-Bench: Benchmarking LLM Agents on Multimodal Data-Driven Urban Diagnosis

Abstract

Urban diagnosis integrates heterogeneous observations to identify urban problems, localize affected areas, and investigate contributing factors, informing evidence-based urban planning and management. However, its reliance on labor-intensive, case-specific expert workflows limits scalability and reuse, motivating the exploration of agent-based execution. To evaluate this capability, we introduce DUDA-Bench, a hierarchical and interactive benchmark that formalizes data-driven urban diagnosis as a multi-stage agent workflow. It comprises 86 atomic and 22 workflow tasks spanning four analytical stages, grounded in multimodal data from 12 cities covering five urban problem types. Evaluations of seven backbone models and five agent systems reveal a substantial gap between isolated analytical competence and end-to-end diagnosis, with system benefits varying across backbones. Trajectory analysis shows that unresolved evidence gaps propagate across stages, while successful recovery involves revising assumptions and actions using feedback. These findings highlight limitations in coordinating analytical capabilities across stages, particularly adaptive planning, evidence integration, and verification. More broadly, DUDA-Bench provides a framework for translating expert analytical workflows into hierarchical agent tasks and process-aware evaluation, supporting systematic assessment of end-to-end analytical capabilities.

Primary area:
datasets and benchmarks
Keywords:
large language model agents, urban computing, benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZCWLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DUDA-Bench formulates data-driven urban diagnosis as a four-stage workflow comprising problem framing, urban profiling, diagnostic reasoning, and validation and synthesis. It combines multimodal observations from 12 cities with 86 atomic tasks and 22 workflow tasks covering five urban problem types. A shared execution environment supports evaluation through issue recognition, localization, diagnostic evidence, and trajectory checkpoints. Experiments compare seven backbone models and five agent systems, finding substantially different workflow scores among models with similar aggregate atomic performance. The main contribution is the domain-specific task formulation, data and execution infrastructure, and process evaluation.

Critical strengths
  1. The benchmark integrates heterogeneous data access, indicator computation, localization, and evidence synthesis within an executable environment, while supporting both atomic and complete-workflow evaluation. This is closer to practical analytical work than isolated question answering or indicator estimation and provides useful infrastructure for studying extended analytical execution.

  2. The 14 checkpoints describe analytical objectives rather than prescribing a tool sequence, allowing different valid execution strategies. Human verification covers 56 trajectories and 716 applicable checkpoint judgments, with independent annotation and third-reviewer adjudication. The reported 93.6% human agreement and 90.4% judge accuracy provide meaningful support for the trajectory evaluator.

Critical weaknesses
  1. Section 3.2 states that workflow references are extracted from published conclusions, whereas atomic reference outputs are computed using the benchmark data and manually checked. Although both task levels are reviewed for data availability and evaluation feasibility, the paper does not explain how literature-derived workflow conclusions are verified against each case’s actual spatial scope and observation period. This matters because the benchmark’s overall coverage is 2015–2024, while the cited Phoenix study analyzes earlier changes, including those between 1992 and 2001. This difference does not establish incorrect labels, but task-level provenance is needed to assess their validity. The outcome metrics introduce related concerns: under Equations (1)–(4), adding unsupported issue families while retaining the same target-specific predictions does not reduce the specified score. Localization emphasizes reference coverage, but the description does not specify how spatial false positives below the applicable 25% area cap are penalized. Evidence-matching rules, including numerical tolerances, spatiotemporal alignment, and valid alternative indicators, are also insufficiently detailed. The human verification in Appendix A.3 supports checkpoint judgments, but the paper does not report direct validation of the headline outcome score against expert assessments of overall diagnostic quality.

  2. Atomic PR averages binary pass outcomes, whereas Workflow Score averages recognition-gated continuous scores; their numerical difference cannot directly quantify a capability gap on a common scale. Equal aggregate Atomic PR also does not establish equal competence on the operations required by each workflow. The checkpoint and trajectory analyses support coordination difficulties as an explanation, but do not isolate them from underlying analytical failures. Workflow-linked atomic analyses or controlled provision of correct intermediate evidence would strengthen this interpretation. The component findings likewise require qualification: although Section 4.1 lists separate removals, Figure 2(B1) evaluates routing and tool removal in the no-code setting. Thus, 13.7% to 12.4% change does not establish routing’s contribution with code available, and interpreting the zero score after removing both code and tools requires specifying what data access remains. Finally, paper do not specify repetition counts, decoding settings. Uncertainty estimates for the main comparisons are also absent. These limitations constrain the interpretation and reproducibility of the reported system differences.

Questions for authors
  1. Section 3.2 states that workflow references are extracted from published conclusions, that atomic references are computed using the benchmark data and manually checked, and that both task levels are reviewed for data availability and evaluation feasibility. Could you clarify how literature-derived workflow conclusions were checked against each case’s actual observation period and spatial scope? A concrete provenance example, including whether any conclusions were revised or excluded after checking the benchmark observations, would help assess reference validity.

  2. Under Equations (1)–(4), do not see an explicit penalty for additional predicted issue families unrelated to an evaluated target. Does the implementation include such a penalty or other relevant output constraints? Please also clarify the spatial matching and evidence-matching rules, including numerical tolerances where applicable. Equation (5) defines NwN_w over workflow targets: what is its actual value, how many targets belong to each of the 22 workflows, and how does this aggregation correspond to the table notes stating that problems are equally weighted? Beyond the checkpoint validation in Appendix A.3, is any expert validation of final outcome scores available?

  3. Could you provide the repetition counts, decoding settings, and instantiated environment-action limits used for each system? Please specify any configured token limits and the exact model/checkpoint where needed to fix the evaluated versions. Are uncertainty estimates for the main comparisons available? For the lightweight LAMBDA adaptation, which original collaboration mechanisms were retained?

  4. Figure 2(B1) evaluates routing and tool removal in the no-code setting. What data access remains after both code execution and domain tools are removed? Are independent component ablations with code retained, workflow-linked atomic analyses, or controlled runs with correct intermediate evidence available? These would help assess whether the component findings extend to the full system and distinguish coordination difficulties from failures of the underlying analytical operations. If such analyses are unavailable, please clarify the intended scope of these conclusions.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZCWL for W1, Q1 (1/4)

Q1: Section 3.2 states that workflow references are extracted from published conclusions, that atomic references are computed using the benchmark data and manually checked, and that both task levels are reviewed for data availability and evaluation feasibility. Could you clarify how literature-derived workflow conclusions were checked against each case’s actual observation period and spatial scope? A concrete provenance example, including whether any conclusions were revised or excluded after checking the benchmark observations, would help assess reference validity.

A1: Thank you for your constructive comments and suggestions. We first compare the observation dates and study area in the source paper with the years and geographic boundary of the benchmark case. We then check the published diagnosis against measurements from the available data within that boundary. We use the Valencia old-town heat-island task to illustrate both checks.

Valencia example. Wei and Sobrino [2] used 17 ECOSTRESS and Landsat images acquired during two summer heatwaves in 2022 and 2023. They compared land surface temperature (LST) across Valencia’s local climate zones and found elevated surface heat in compact built-up areas. Both summers fall within the 2015–2024 benchmark window. We use this finding to define an open workflow task for Valencia old town. The agent receives regional Landsat LST, land cover, and imagery for a fixed bounding box [-0.40, 39.45, -0.34, 39.50]. It considers all five issue families and reports a diagnosis, affected locations, and supporting evidence. For reference, we have submitted the Valencia case in the supplementary material. We also provide the benchmark implementation and part of the data acquisition code. The complete tasks and dataset will be released upon acceptance.

Checking the Valencia reference against the observations. Experts mapped the published finding to three reference components: surface urban heat island as the issue family, Valencia old town as the target area, and summer LST, Gi* hotspots, low vegetation cover, and impervious-surface fraction as diagnostic indicators. We then computed the atomic references on the benchmark data for that same area. The latest summer Landsat raster gives a mean LST of 317.567 K and a 95th percentile of 319.942 K. The hottest cell in the 4 × 4 grid is (3,1), counting from zero at the northwest corner. Overlaying the high-LST pixels with WorldCover shows that 97.35% of pixels above the 90th temperature percentile fall on non-green land. This checks the link between heat and land cover within the task boundary. In the workflow evaluation, the agent must identify the issue, locate it, and report indicators that match the references. The supplementary material includes the complete Valencia case.

Thank you for pointing out the mismatch between the dates of the cited Phoenix study and the benchmark observation period. The study of land-cover change between 1992 and 2001 provided the problem background and analytical basis for the Phoenix task. The ground truth for the urban sprawl diagnosis comes from Fan et al. [1]. Our original manuscript omitted this ground-truth reference. Fan et al. mapped land-cover changes across metropolitan Phoenix using observations from 2001, 2011, and 2021. Their 2021 map falls within the benchmark window of 2015–2024 and documents where agricultural and other land had been converted to developed land. The study therefore supports the diagnosis through 2021.

References

[1] C. Fan, M. J. I. J. Prothan, Y. Zhu, and D. Shi, “Two Decades of Urban Transformation and Heat Dynamics in a Desert Metropolis: Linking Land Cover, Demographics, and Surface Temperature,” Land, vol. 14, no. 11, Art. no. 2141, 2025. https://doi.org/10.3390/land14112141

[2] L. Wei and J. A. Sobrino, “Surface urban heat island analysis based on local climate zones using ECOSTRESS and Landsat data: A case study of Valencia city (Spain),” International Journal of Applied Earth Observation and Geoinformation, vol. 130, Art. no. 103875, 2024. https://doi.org/10.1016/j.jag.2024.103875

AuthorsOpenReview ↗

Response to Reviewer ZCWL for W1, Q2 (2/4)

Q2: Under Equations (1)–(4), do not see an explicit penalty for additional predicted issue families unrelated to an evaluated target. Does the implementation include such a penalty or other relevant output constraints? Please also clarify the spatial matching and evidence-matching rules, including numerical tolerances where applicable. Equation (5) defines (N_w) over workflow targets: what is its actual value, how many targets belong to each of the 22 workflows, and how does this aggregation correspond to the table notes stating that problems are equally weighted? Beyond the checkpoint validation in Appendix A.3, is any expert validation of final outcome scores available?

A2: Additional issue families. The current metric does not directly penalize extra predicted issue families. The reference annotations for each task record the target issues supported by evidence. Different urban issues can coexist in the same area over the same period [1]. For example, reporting both urban sprawl and urban shrinkage does not by itself imply a contradiction, because population and built-up area are different indicators. A study of 188 European cities found that built-up land could expand even as the population declined in the same city over the same period [1]. If a task is annotated only for urban shrinkage, treating an additional diagnosis of urban sprawl as a false positive solely on that basis could penalize a valid diagnosis. Even if an agent reports all five issue families, only those that match the task’s reference annotations count as hits. We acknowledge that this design also leaves unsupported extra diagnoses without a direct penalty.

We plan to add a consistency check to Layer A. If the final report makes incompatible claims about the same indicator in the same area over the same period and leaves the contradiction unresolved, any diagnosis that relies on those claims receives A=0A=0. For example, a report might claim that the population of the same area both increased and decreased between 2015 and 2020.

Matching predictions to references. Experts extract three pieces of reference information from each source study: the diagnosed issue, its location, and the indicators that support the diagnosis. In Valencia, the reference issue is a surface urban heat island, and the reference location is the old town. The reference indicators are summer land surface temperature, Gi* hotspots, low vegetation cover, and impervious surface fraction. Equations (1)–(4) match each diagnosis that the agent explicitly supports in its final report against these references.

  • Recognition (A). If the report explicitly diagnoses a surface urban heat island, the target receives A = 1. If it lists the heat island only as a candidate for further investigation, A = 0, and B, C, and the Workflow Score are also zero for that target.
  • Localization (B). Only locations linked to the diagnosis are scored. For region tasks, place names or equivalent descriptions are matched to reference areas. B is the fraction of reference areas matched. For city tasks, reported locations are mapped to geographic areas. B is the fraction of the reference area covered by the prediction. If the predicted area exceeds 25% of the city bounding box, B = 0. Within this limit, extra predicted area carries no separate penalty.
  • Indicator F1 (C). Indicators used to support the diagnosis are matched to reference indicators by meaning. Equivalent terms count once. For example, if the report gives only summer land surface temperature and impervious surface fraction, it matches two of the four Valencia reference indicators. Thus, C = 2 × 2 / (2 + 4) = 2/3. Both extra and missing indicators lower F1. A proxy with a different meaning can provide supplementary evidence but does not automatically count as a match. Numerical atomic tasks are scored against task-specific reference values and tolerances.

In Equation (5), Nw=22N_w=22. Each of the 22 workflow reference annotations corresponds to one target. When several targets share the same input, each answer to that input is scored against each target. We first average the scores within each target and then take the equally weighted mean across all 22 targets. The expert validation in Appendix A.3 covers 56 trajectories and 716 applicable checkpoint judgments.

Expert Validation. Expert involvement is integrated into the benchmark construction. Both the ground-truth references and outcome evaluation criteria were manually developed and reviewed by domain experts. Beyond the checkpoint-level validation described in Appendix A.3, we additionally conducted expert spot-checks of sampled final outcomes to assess the reliability of the reported scores.

Reference

[1] Haase, Kabisch & Haase (2013). Endless Urban Growth? On the Mismatch of Population, Household and Urban Land Area Growth and Its Effects on the Urban Debate. PLOS ONE.

AuthorsOpenReview ↗

Response to Reviewer ZCWL for W2, Q3 (3/4)

Q3: Could you provide the repetition counts, decoding settings, and instantiated environment-action limits used for each system? Please specify any configured token limits and the exact model/checkpoint where needed to fix the evaluated versions. Are uncertainty estimates for the main comparisons available? For the lightweight LAMBDA adaptation, which original collaboration mechanisms were retained?

A3: System settings. The five agent systems evaluated in the paper share a budget of 25 environment actions per workflow in their latest configurations. Each executed tool or code action counts toward this budget. The Qwen runs use temperature 0.2. The latest configurations set no client-side output-token cap.

  • Direct uses two model turns. It selects a batch of actions and then synthesizes the observations.
  • Plan-and-Execute uses two model turns. It fixes the plan before execution and synthesizes the results afterward. The executor allows up to two worker retries.
  • ReAct selects one action at a time and observes the result before selecting the next, up to the shared 25-action limit. It allows five protocol errors and two finalization attempts.
  • MAF-Magentic allows 40 model calls across up to 15 coordination rounds. The higher call limit accommodates the Manager and Reviewer calls in addition to the Analyst’s environment actions.
  • LAMBDA allows 27 model calls for up to 25 environment actions, leaving room for final synthesis. As described in Appendix C.5, the original multi-agent system uses an interactive interface and external execution components. DUDA-Bench evaluates all systems in a common execution environment. We therefore followed the sequential tool-execution setup of EvoDS[1]. Our adaptation retains LAMBDA’s iterative analysis loop. The agent selects a function, observes its result, and uses that feedback to choose its next action. A single agent runs this loop in the benchmark.

Repetitions and uncertainty. The original comparisons used one run per configuration. To measure run-to-run variation, we repeated all 22 ReAct workflows twice for each of qwen3.8-max and qwen3.8-27b under the same 25-action budget. Across these two repetitions, the mean Workflow Score was 23.8% for Max and 23.4% for 27B. The sample standard deviations were 3.6 and 2.5 percentage points, respectively. Table R1 also gives the original scores and the three-run means.

Table R1. Workflow outcomes by model and run.

ModelRunRecognition (A) ↑Localization (B) ↑Indicator F1 (C) ↑Workflow Score ↑
Qwen3.8-MaxOriginal0.7270.2740.2810.277
Repeat 10.5450.2470.1780.213
Repeat 20.6360.3080.2200.264
Three-run mean0.6360.2760.2270.251
Qwen3.8-27BOriginal0.4320.1250.1410.133
Repeat 10.6140.2840.2190.252
Repeat 20.5910.2190.2130.216
Three-run mean0.5450.2090.1910.200

Reference [1] Z. Yang, F. Liu, Y. Ning, and H. Liu, “EvoDS: Self-Evolving Autonomous Data Science Agent with Skill Learning and Context Management,” in Proc. ACM SIGKDD (KDD), 2026, pp. 6128–6139.

AuthorsOpenReview ↗

Response to Reviewer ZCWL for W2, Q4 (4/4)

Q4: Figure 2(B1) evaluates routing and tool removal in the no-code setting. What data access remains after both code execution and domain tools are removed? Are independent component ablations with code retained, workflow-linked atomic analyses, or controlled runs with correct intermediate evidence available? These would help assess whether the component findings extend to the full system and distinguish coordination difficulties from failures of the underlying analytical operations. If such analyses are unavailable, please clarify the intended scope of these conclusions.

A4: Independent component ablations with code retained. Figure 2(B1) removes skill routing or domain tools after disabling code execution. With both code execution and domain tools disabled, the agent receives descriptive metadata in the data card and has no way to inspect the underlying data. In the new ablations, we retained code execution and removed either skill routing alone or the analytical and visual tools alone. The full system and both ablations used the same budget of 25 environment actions.

Table R2. Independent ablations with code retained (Qwen3.8-Max).

ModelConditionRecognition (A) ↑Localization (B) ↑Indicator F1 (C) ↑Workflow Score ↑
Qwen3.8-MaxFull (mean of 2 runs)0.5910.2780.1990.238
w/o skill routing (1 run)0.5910.2830.2080.245
w/o tools (1 run)0.6820.4310.2330.332

With analytical and visual tools removed, the agent could still inspect data and compute indicators in Python. Across the 22 workflows, the trace records 471 code actions in this condition, compared with 302 in each full-system run. The largest score increase was in localization (0.278 to 0.431). These observations suggest that the agent used code to perform more of the analysis itself. With one run for this ablation, the improvement cannot be attributed to tool removal alone.

Workflow-linked atomic analyses. We matched all 22 workflows in the original Qwen3.8-Max ReAct run to their constituent atomic tasks. Each of the 15 workflows that failed the strict success criterion had at least one atomic task the agent solved in isolation. Of their 56 atomic tasks, 26 passed (46.4%) and 30 failed (53.6%). In the Mumbai sprawl case, the agent correctly identified the direction of urban expansion in the atomic task. In the paired workflow, it recognized sprawl and retrieved grid coordinates at step 12 that placed row 1 at the southern edge of the study area. Its final report nevertheless described cells r1c3 and r1c4 as the “NW periphery.” This shows where coordination broke down when the agent translated the coordinates into its final diagnosis.

Reviewer_fhiwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DUDA-Bench, a benchmark to evaluate agent performance for urban diagnosis as a multi-stage agent workflow, including problem framing, urban profiling, diagnostic reasoning, and validation/synthesis. Its main contributions are a design of 86 atomic tasks and 22 workflow tasks, a dataset covering 12 cities and five urban problem types, and an interactive environment with domain tools, reusable skills, and code execution. Evaluation includes atomic validators, workflow scores, and 14 trajectory checkpoints. Experiments evaluate seven backbones under ReAct and five agent systems on two backbones, as well as component ablations and trajectory analysis. Atomic pass rates range from 0.419 to 0.558, the workflow scores range from 0.071 to 0.277. The authors attribute workflow difficulties to evidence acquisition and cross-stage coordination. Human verification of 716 checkpoint judgments from 56 trajectories gives a 90.4% judge accuracy against adjudicated labels.

Critical strengths
  1. Combining atomic tasks with workflows, heterogeneous data sources, and executable environment makes this benchmark a potentially valuable evaluation resource. The benchmark could expose failures that individual GIS operations miss, while retaining component tasks that help investigate those failures.

  2. The evaluation examines intermediate analytical quality and validates its trajectory judgment. The checkpoints can assess evidence acquisition, integration, localization, explanatory reasoning, and uncertainty. The authors also used independent annotations and adjudication. Together with the backbone and system comparisons, it makes the benchmark potentially useful for diagnosing agent limitations, not just producing rankings.

Critical weaknesses
  1. The paper describes manual extraction by two experts, but didn't provide a complete task-and-reference example or an operational annotation protocol. In particular, it is unclear how published studies are translated into task instructions, reference locations, and required diagnostic evidence. It is difficult to determine what agents are being asked to solve and what constitutes a correct solution. Because the benchmark is the primary contribution, this limits confidence in its reproducibility and the interpretation of the reported scores.

  2. The claimed coordination bottleneck or individual component benefits are not adequately supported by experiments. Atomic PR is a binary success rate over individual operations, while Workflow Score is a recognition-gated average of localization and evidence scores. Only their numerical difference cannot quantify a coordination deficit because different scoring rules and task demands also contribute. The equal atomic scores but different workflow scores in Table 2 are suggestive, yet there is no matched analysis showing that operations an agent solves independently fail when embedded in their corresponding workflows. In addition, for the ablation study, "Figure 2(B1) Component Ablation" evaluates “w/o code,” “w/o code + skill-routing,” and “w/o code + tools”. It measures the latter components conditionally on code already being absent, rather than their independent contribution to the full system.

Questions for authors
  1. Could you provide one complete workflow example, including the source study, the exact prompt given to the agent, the available data, the expected output format, and the reference issue labels, locations, and diagnostic evidence used for scoring? Please show how the study was translated into this task and reference answer, and how its constituent atomic tasks were derived.

  2. What criteria did the two experts use when selecting studies, defining tasks, and extracting reference diagnoses?

  3. How are an agent's reported locations and diagnostic evidence matched to the references, including spatial criteria, numerical tolerances, and valid alternative indicators? How are incorrect extra issue families handled? Under the current Recognition rule, predicting all five families would satisfy recognition for every target, is there an additional penalty?

  4. Is there a task-matched analysis linking each workflow to its constituent atomic tasks and checking whether agents fail operations they can solve in isolation? If so, does it support the coordination interpretation after accounting for the different scoring rules?

  5. Were skill routing and domain tools also removed individually from the full system, or only with code execution already disabled, as Figure 2(B1) indicates? Please also explain why the full-system score in this figure is 28.5%, compared with 27.7% for Qwen3.8-Max with ReAct in Table 2.

3 comments
AuthorsOpenReview ↗

Response to Reviewer fhiw for W1, Q1 and Q2 (1/3)

Q1: Could you provide one complete workflow example, including the source study, the exact prompt given to the agent, the available data, the expected output format, and the reference issue labels, locations, and diagnostic evidence used for scoring? Please show how the study was translated into this task and reference answer, and how its constituent atomic tasks were derived.

A1: Thank you very much for your constructive comments and suggestions. We use the Valencia old-town heat-island task as a complete workflow example. Experts first extracted the link between built form, vegetation cover, and surface temperature [1]. They translated this finding into an open task: use the available multimodal data for Valencia old town over 2015–2024 to identify urban problems, locate affected areas, explain mechanisms, and derive evidence-based planning implications. The prompt lists all five issue families without revealing the reference diagnosis.

Available data include regional Landsat LST, WorldCover, satellite imagery, and the OSM and street-view data listed in the data card. The spatial scope is fixed to the bounding box [-0.40, 39.45, -0.34, 39.50]. The expected output is one JSON object containing the problem representation, indicator evidence, diagnosis, mechanisms, uncertainty, and planning implications. Claims must identify their supporting evidence, spatial units, and observation years.

Experts aligned the ground truth with the three evaluation layers, including heat island as the issue family, the geographic area covering Valencia old town as the reference location, and summer LST, Gi* hotspots, low vegetation cover, and impervious-surface fraction as diagnostic evidence. The source study provides the diagnosis and mechanism. Experts then translate these into geographic units and measurable indicators for the benchmark.

When constructing atomic tasks, we defined four independently verifiable operations: (1) the overlap between high-LST pixels and non-green land cover (reference: 0.973495); (2) the latest-year mean and 95th-percentile LST (317.567 K and 319.942 K); (3) the hottest cell in a 4×4 grid (cell (3,1), with zero-based rows from north to south and columns from west to east); and (4) code that computes the mean of the hottest 5% of pixels minus the overall mean (3.613 K). These reference outputs are computed from the benchmark data, rather than copied from historical measurements.

For reference, we have submitted the Valencia case in the supplementary material. We also provide the benchmark implementation and part of the data acquisition code. The complete tasks and dataset will be released upon acceptance.

Reference

[1] Wei, L., & Sobrino, J. A. (2024). Surface urban heat island analysis based on local climate zones using ECOSTRESS and Landsat data: A case study of Valencia city (Spain). International Journal of Applied Earth Observation and Geoinformation, 130, 103875.

Q2: What criteria did the two experts use when selecting studies, defining tasks, and extracting reference diagnoses?

A2: Manual annotation consists of (1) study selection, (2) conclusion extraction, (3) task definition, and (4) data verification. Experts select studies with an explicit target city, urban problem, and spatial pattern. They extract diagnoses, locations, indicators, and mechanisms from the results and figures. They map place names to areas of interest and give indicators computable definitions. They then review data coverage, observation years, and quality to assess evaluation feasibility. Experts distinguish published conclusions, benchmark proxy indicators, and explanatory inferences. For numerical ground truth, answers are recomputed on the DUDA-Bench dataset to avoid treating historical values as exact answers for the benchmark observations. The two experts cross-check diagnoses, spatial scope, and evidence definitions, and resolve disagreements through discussion.

AuthorsOpenReview ↗

Response to Reviewer fhiw for Q3 (2/3)

Q3: How are an agent's reported locations and diagnostic evidence matched to the references, including spatial criteria, numerical tolerances, and valid alternative indicators? How are incorrect extra issue families handled? Under the current Recognition rule, predicting all five families would satisfy recognition for every target, is there an additional penalty?

A3: Matching predictions to references. Experts extract three pieces of reference information from each source study: the diagnosed issue, its location, and the indicators that support the diagnosis. In Valencia, the reference issue is a surface urban heat island, and the reference location is the old town. The reference indicators are summer land surface temperature, Gi* hotspots, low vegetation cover, and impervious surface fraction. Equations (1)–(4) match each diagnosis that the agent explicitly supports in its final report against these references.

  • Recognition (A). If the report explicitly diagnoses a surface urban heat island, the target receives A = 1. If it lists the heat island only as a candidate for further investigation, A = 0, and B, C, and the Workflow Score are also zero for that target.
  • Localization (B). Only locations linked to the diagnosis are scored. For region tasks, place names or equivalent descriptions are matched to reference areas. B is the fraction of reference areas matched. For city tasks, reported locations are mapped to geographic areas. B is the fraction of the reference area covered by the prediction. If the predicted area exceeds 25% of the city bounding box, B = 0. Within this limit, extra predicted area carries no separate penalty.
  • Indicator F1 (C). Indicators used to support the diagnosis are matched to reference indicators by meaning. Equivalent terms count once. For example, if the report gives only summer land surface temperature and impervious surface fraction, it matches two of the four Valencia reference indicators. Thus, C = 2 × 2 / (2 + 4) = 2/3. Both extra and missing indicators lower F1. A proxy with a different meaning can provide supplementary evidence but does not automatically count as a match. Numerical atomic tasks are scored against task-specific reference values and tolerances.

Additional issue families. The current metric does not directly penalize extra predicted issue families. The reference annotations for each task record the target issues supported by evidence. Different urban issues can coexist in the same area over the same period [1]. For example, reporting both urban sprawl and urban shrinkage does not by itself imply a contradiction, because population and built-up area are different indicators. A study of 188 European cities found that built-up land could expand even as the population declined in the same city over the same period [1]. If a task is annotated only for urban shrinkage, treating an additional diagnosis of urban sprawl as a false positive solely on that basis could penalize a valid diagnosis. Even if an agent reports all five issue families, only those that match the task’s reference annotations count as hits. We acknowledge that this design also leaves unsupported extra diagnoses without a direct penalty.

We plan to add a consistency check to Layer A. If the final report makes incompatible claims about the same indicator in the same area over the same period and leaves the contradiction unresolved, any diagnosis that relies on those claims receives A=0A=0. For example, a report might claim that the population of the same area both increased and decreased between 2015 and 2020.

[1] Haase, Kabisch & Haase (2013). Endless Urban Growth? On the Mismatch of Population, Household and Urban Land Area Growth and Its Effects on the Urban Debate. PLOS ONE.

AuthorsOpenReview ↗

Response to Reviewer fhiw for W2, Q4–Q5 (3/3)

Q4: Is there a task-matched analysis linking each workflow to its constituent atomic tasks and checking whether agents fail operations they can solve in isolation? If so, does it support the coordination interpretation after accounting for the different scoring rules?

A4: We matched all 22 workflows to their constituent atomic tasks in the original Qwen3.8-Max ReAct run. Fifteen workflows failed the strict success criterion. Each had at least one atomic task that the agent solved in isolation. Of the 56 atomic tasks linked to these failed workflows, 26 passed (46.4%) and 30 failed (53.6%).

For Mumbai sprawl, we compared the agent’s result on the expansion-direction atomic task with its account of expansion in the paired workflow. In isolation, the agent correctly identified the direction of urban expansion. In the workflow, it recognized sprawl and retrieved the grid coordinates at step 12. Those coordinates place row 1 at the southern edge of the study area, yet the final report described cells r1c3 and r1c4 as the “NW periphery.” This error illustrates the coordination difficulty identified in our trajectory analysis: the agent obtained the spatial evidence but used it incorrectly in its final diagnosis.

Q5: Were skill routing and domain tools also removed individually from the full system, or only with code execution already disabled, as Figure 2(B1) indicates? Please also explain why the full-system score in this figure is 28.5%, compared with 27.7% for Qwen3.8-Max with ReAct in Table 2.

A5: We thank the reviewer for pointing out this inconsistency. We found that the original ablation experiments used a budget of 35 turns. Results from a single run can also vary by chance. We corrected this issue in the new ablation experiments. All runs in this comparison used a budget of 25 turns.

The original combined ablations removed routing or tools with code execution already disabled. They therefore measure conditional contributions. We added two independent component ablations with code retained: removing only skill routing, or removing only analytical and visual tools. Both use a limit of 25 environment actions.

Table R1. Independent ablations with code retained (Qwen3.8-Max).

ModelConditionRecognition (A) ↑Localization (B) ↑Indicator F1 (C) ↑Workflow Score ↑
Qwen3.8-MaxFull (mean of 2 runs)0.5910.2780.1990.238
w/o skill routing (1 run)0.5910.2830.2080.245
w/o tools (1 run)0.6820.4310.2330.332

With analytical and visual tools removed, the agent could still inspect data and compute indicators in Python. Across the 22 workflows, the trace records 471 code actions in this condition, compared with 302 in each full-system run. The largest score increase was in localization (0.278 to 0.431). These observations suggest that the agent used code to perform more of the analysis itself. With one run for this ablation, the improvement cannot be attributed to tool removal alone.

Reviewer_Wn8TOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work propose DUDA-bench, a benchmark for evaluating whether LLM agents can integrate heterogeneous urban evidence into complete diagnostic workflows, beyond performing isolated analytical operations. It comprises 86 atomic and 22 workflow tasks spanning four analytical stages, grounded in multimodal data from 12 cities covering five urban problem types. Authors deduct 7 backbones and 5 agents for evaluation, highlight limitations of current LLM-based agent systems in coordinating analytical capabilities across stages, particularly adaptive planning, evidence integration, and verification

Critical strengths
  1. This work propose a clear workflow for benchmark construction and evaluation.
  2. This work propose a benchmark considering a wider data converage.
Critical weaknesses
  1. Extremely narrow problem taxonomy. This work only focus on 5 types of urban research, which are not the core research questions. Several related works also propose benchmarks, focus on variants of core questions, including congestion forecasting, OD/mobility prediction, next-POI prediction, traffic signal control and route planning (USTBench, ICLR 2026), and mobility prediction, traffic control and urban navigation (CityBench, KDD 2025). However, these questions do not get studied in this work, makes this benchmarks unable to verify the capacity of urban agents fairly.
  2. This benchmark only focus on statistical urban data, lack of dynamic human-mobility data. These data cannot reflect how real human travel and act in real urban environments. Although the author propose "temporal data", which are actually annual snapshots over 2015–2024, and the only temporal operations offered are annual means and year-over-year change rates (Table 9). These kinds of data have a strong gap between real-world urban challenges.
Questions for authors
  1. Why do authors think the capacities you benchmarked are key challenges and research questions of urban reseaarch?
  2. How does the evaluation penalize unsupported additional diagnoses? Have the authors tested a baseline that predicts all five issue families?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Wn8T on W1–W2/Q1 (1/2)

W1. Extremely narrow problem taxonomy. This work only focus on 5 types of urban research, which are not the core research questions. Several related works also propose benchmarks, focus on variants of core questions, including congestion forecasting, OD/mobility prediction, next-POI prediction, traffic signal control and route planning (USTBench, ICLR 2026), and mobility prediction, traffic control and urban navigation (CityBench, KDD 2025). However, these questions do not get studied in this work, makes this benchmarks unable to verify the capacity of urban agents fairly.

Q1. Why do authors think the capacities you benchmarked are key challenges and research questions of urban reseaarch?

A1: Thank you for your constructive comments and suggestions. Urban diagnosis directly informs planning by identifying problems, locating affected areas, and using evidence to guide interventions. Previous studies have applied this process to community renewal [1], urban shrinkage [2], and the allocation of public services [3]. These studies typically focus on a particular problem or region. To our knowledge, DUDA-Bench is the first benchmark to bring open-ended diagnosis across multiple urban problem types into a unified framework for evaluating agents. It assesses whether agents can independently select evidence, locate problems, and substantiate their diagnoses, going beyond individual, predefined analyses.

As discussed in Table 1, section2, and Appendix B, CityBench, USTBench, and DUDA-Bench have different evaluation goals. CityBench and USTBench cover urban perception, spatiotemporal prediction, and interactive decision-making. By contrast, DUDA-Bench focuses on the full diagnostic process of determining what problems are present, where they occur, and what evidence supports each diagnosis. We recognize the importance of prediction, control, and navigation. Our benchmark complements these efforts by evaluating urban diagnosis, without claiming to cover the full range of urban agent capabilities. Appendix B provides a detailed comparison.

W2. This benchmark only focus on statistical urban data, lack of dynamic human-mobility data. These data cannot reflect how real human travel and act in real urban environments. Although the author propose "temporal data", which are actually annual snapshots over 2015–2024, and the only temporal operations offered are annual means and year-over-year change rates (Table 9). These kinds of data have a strong gap between real-world urban challenges.

A2: We selected five problems spanning three dimensions of urban planning. In urban morphology, diagnosing sprawl helps planners identify habitats threatened by land development [4], while diagnosing shrinkage reveals where population loss and housing vacancy are concentrated [2]. In ecology and climate resilience, identifying urban heat islands helps prioritize areas for cooling through greening [5], while assessing green-space fragmentation informs habitat protection and connectivity [6]. In service accessibility, comparing facility provision with local demand reveals which communities are underserved [3].

We chose data at the spatial and temporal scales these questions require. Satellite imagery, street views, and maps provide spatial evidence, while observations of built-up area and population across years help agents assess longer-term change. We agree that annual observations cannot resolve daily travel patterns. DUDA-Bench focuses on urban conditions and how they change across neighborhoods and cities. Human-mobility tasks focus on when, where, and how people travel. For example, our accessibility tasks use population and facility locations to assess which neighborhoods have limited potential access to services. Determining which facilities residents actually visit requires trip or visit records.

References

[1] Ye et al. Community Renewal with Urban Diagnosis: Bajiao Community, Shijingshan District, Beijing. ISOCARP World Planning Congress, 2021.
[2] Xie et al. Examining Shrinking City of Detroit in the Context of Socio-Spatial Inequalities. Landscape and Urban Planning, 2018.
[3] Kang and Lee. Assessing Accessibility and Equity in Childcare Facilities Through 2SFCA: Insights from Housing Types in Seongbuk-gu, Seoul. ISPRS International Journal of Geo-Information, 2025.
[4] Seto et al. Global Forecasts of Urban Expansion to 2030 and Direct Impacts on Biodiversity and Carbon Pools. PNAS, 2012.
[5] Li et al. Green Spaces Provide Substantial but Unequal Urban Cooling Globally. Nature Communications, 2024.
[6] Beninde et al. Biodiversity in Cities Needs Space: A Meta-analysis of Factors Determining Intra-urban Biodiversity Variation. Ecology Letters, 2015.

AuthorsOpenReview ↗

Response to Reviewer Wn8T on Q2 (2/2)

Q2. How does the evaluation penalize unsupported additional diagnoses? Have the authors tested a baseline that predicts all five issue families?

A3: The current metric does not directly penalize extra predicted issue families. The reference annotations for each task record the target issues supported by evidence. Research shows that different urban issues can coexist in the same area over the same period [1]. For example, reporting both urban sprawl and urban shrinkage does not by itself imply a contradiction, because population and built-up area are different indicators. A study of 188 European cities found that built-up land could expand even as the population declined in the same city over the same period [1]. Therefore, if a task is annotated only for urban shrinkage, treating an additional diagnosis of urban sprawl as a false positive solely on that basis could penalize a valid diagnosis. Even if an agent reports all five issue families, only those that match the task’s reference annotations count as hits. We acknowledge that this design also leaves unsupported extra diagnoses without a direct penalty.

We plan to add a consistency check to Layer A. If the final report makes two incompatible claims about the same indicator in the same area over the same period and leaves the contradiction unresolved, any diagnosis that relies on those claims receives A = 0. For example, a report might claim that the population of the same area both increased and decreased between 2015 and 2020.

Reference

[1] Haase, D., Kabisch, N., and Haase, A. (2013). Endless Urban Growth? On the Mismatch of Population, Household and Urban Land Area Growth and Its Effects on the Urban Debate. PLOS ONE, 8(6), e66531. https://doi.org/10.1371/journal.pone.0066531

Comments on the paper

1 comment
AuthorsOpenReview ↗

DUDA-Bench: Summary of Revisions and Author Responses

Dear Program Chairs, Area Chairs, and Reviewers,

Thank you for your constructive feedback. We have revised the paper and highlighted the changes in blue. We summarize the main revisions and findings below.

  • Task construction and reference validity (fhiw W1/Q1–Q2, ZCWL W1/Q1). We clarified how two domain experts select studies, define tasks, and check reference diagnoses against the available observations. We used the Valencia old-town heat-island example to show how the source study informed the agent prompt, input data, reference location, and diagnostic indicators, and how we derived four atomic tasks from the workflow. The supplementary material provides the detailed case.

  • Outcome scoring and extra diagnoses (fhiw Q3, ZCWL W1/Q2, Wn8T Q2). We clarified how recognition, localization, and indicator matching work and how scores are averaged across 22 equally weighted workflow targets. We also explained why extra issue families receive neither credit nor a direct penalty.

  • Atomic tasks and workflow failures (fhiw W2/Q4, ZCWL W2/Q4). We linked all 22 workflows to their constituent atomic tasks for Qwen3.8-Max with ReAct. Across the 15 workflows that failed the strict success criterion, the agent passed 26 of 56 linked atomic tasks in isolation. The Mumbai case shows how coordination can break down: the agent correctly identified the direction of expansion in isolation and retrieved the correct grid coordinates during the workflow, yet described southern cells as northwestern in its final report.

  • Independent ablations and reproducibility (fhiw W2/Q5, ZCWL W2/Q3–Q4). We added independent ablations for skill routing and domain tools, with code execution available under the same 25-action budget. We also added two repeat runs of all 22 workflows for each of two models and documented the execution settings for each agent system.

  • Benchmark scope (Wn8T W1–W2/Q1). We clarified how DUDA-Bench differs in scope from CityBench and USTBench. Its five issue families assess whether agents can identify an urban problem, locate it, and support the diagnosis with evidence. We explained how the data reflect these questions. For example, population and facility locations support analysis of potential service access, while actual travel behavior requires trip or visit records.

Code and data availability. Alongside the Valencia case, we have submitted the benchmark implementation code in the supplementary material. We will release the full task collection and dataset upon acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.