- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
DUDA-Bench formulates data-driven urban diagnosis as a four-stage workflow comprising problem framing, urban profiling, diagnostic reasoning, and validation and synthesis. It combines multimodal observations from 12 cities with 86 atomic tasks and 22 workflow tasks covering five urban problem types. A shared execution environment supports evaluation through issue recognition, localization, diagnostic evidence, and trajectory checkpoints. Experiments compare seven backbone models and five agent systems, finding substantially different workflow scores among models with similar aggregate atomic performance. The main contribution is the domain-specific task formulation, data and execution infrastructure, and process evaluation.
Critical strengths
-
The benchmark integrates heterogeneous data access, indicator computation, localization, and evidence synthesis within an executable environment, while supporting both atomic and complete-workflow evaluation. This is closer to practical analytical work than isolated question answering or indicator estimation and provides useful infrastructure for studying extended analytical execution.
-
The 14 checkpoints describe analytical objectives rather than prescribing a tool sequence, allowing different valid execution strategies. Human verification covers 56 trajectories and 716 applicable checkpoint judgments, with independent annotation and third-reviewer adjudication. The reported 93.6% human agreement and 90.4% judge accuracy provide meaningful support for the trajectory evaluator.
Critical weaknesses
-
Section 3.2 states that workflow references are extracted from published conclusions, whereas atomic reference outputs are computed using the benchmark data and manually checked. Although both task levels are reviewed for data availability and evaluation feasibility, the paper does not explain how literature-derived workflow conclusions are verified against each case’s actual spatial scope and observation period. This matters because the benchmark’s overall coverage is 2015–2024, while the cited Phoenix study analyzes earlier changes, including those between 1992 and 2001. This difference does not establish incorrect labels, but task-level provenance is needed to assess their validity. The outcome metrics introduce related concerns: under Equations (1)–(4), adding unsupported issue families while retaining the same target-specific predictions does not reduce the specified score. Localization emphasizes reference coverage, but the description does not specify how spatial false positives below the applicable 25% area cap are penalized. Evidence-matching rules, including numerical tolerances, spatiotemporal alignment, and valid alternative indicators, are also insufficiently detailed. The human verification in Appendix A.3 supports checkpoint judgments, but the paper does not report direct validation of the headline outcome score against expert assessments of overall diagnostic quality.
-
Atomic PR averages binary pass outcomes, whereas Workflow Score averages recognition-gated continuous scores; their numerical difference cannot directly quantify a capability gap on a common scale. Equal aggregate Atomic PR also does not establish equal competence on the operations required by each workflow. The checkpoint and trajectory analyses support coordination difficulties as an explanation, but do not isolate them from underlying analytical failures. Workflow-linked atomic analyses or controlled provision of correct intermediate evidence would strengthen this interpretation. The component findings likewise require qualification: although Section 4.1 lists separate removals, Figure 2(B1) evaluates routing and tool removal in the no-code setting. Thus, 13.7% to 12.4% change does not establish routing’s contribution with code available, and interpreting the zero score after removing both code and tools requires specifying what data access remains. Finally, paper do not specify repetition counts, decoding settings. Uncertainty estimates for the main comparisons are also absent. These limitations constrain the interpretation and reproducibility of the reported system differences.
Questions for authors
-
Section 3.2 states that workflow references are extracted from published conclusions, that atomic references are computed using the benchmark data and manually checked, and that both task levels are reviewed for data availability and evaluation feasibility. Could you clarify how literature-derived workflow conclusions were checked against each case’s actual observation period and spatial scope? A concrete provenance example, including whether any conclusions were revised or excluded after checking the benchmark observations, would help assess reference validity.
-
Under Equations (1)–(4), do not see an explicit penalty for additional predicted issue families unrelated to an evaluated target. Does the implementation include such a penalty or other relevant output constraints? Please also clarify the spatial matching and evidence-matching rules, including numerical tolerances where applicable. Equation (5) defines over workflow targets: what is its actual value, how many targets belong to each of the 22 workflows, and how does this aggregation correspond to the table notes stating that problems are equally weighted? Beyond the checkpoint validation in Appendix A.3, is any expert validation of final outcome scores available?
-
Could you provide the repetition counts, decoding settings, and instantiated environment-action limits used for each system? Please specify any configured token limits and the exact model/checkpoint where needed to fix the evaluated versions. Are uncertainty estimates for the main comparisons available? For the lightweight LAMBDA adaptation, which original collaboration mechanisms were retained?
-
Figure 2(B1) evaluates routing and tool removal in the no-code setting. What data access remains after both code execution and domain tools are removed? Are independent component ablations with code retained, workflow-linked atomic analyses, or controlled runs with correct intermediate evidence available? These would help assess whether the component findings extend to the full system and distinguish coordination difficulties from failures of the underlying analytical operations. If such analyses are unavailable, please clarify the intended scope of these conclusions.
4 comments
Response to Reviewer ZCWL for W1, Q2 (2/4)
Q2: Under Equations (1)–(4), do not see an explicit penalty for additional predicted issue families unrelated to an evaluated target. Does the implementation include such a penalty or other relevant output constraints? Please also clarify the spatial matching and evidence-matching rules, including numerical tolerances where applicable. Equation (5) defines (N_w) over workflow targets: what is its actual value, how many targets belong to each of the 22 workflows, and how does this aggregation correspond to the table notes stating that problems are equally weighted? Beyond the checkpoint validation in Appendix A.3, is any expert validation of final outcome scores available?
A2: Additional issue families. The current metric does not directly penalize extra predicted issue families. The reference annotations for each task record the target issues supported by evidence. Different urban issues can coexist in the same area over the same period [1]. For example, reporting both urban sprawl and urban shrinkage does not by itself imply a contradiction, because population and built-up area are different indicators. A study of 188 European cities found that built-up land could expand even as the population declined in the same city over the same period [1]. If a task is annotated only for urban shrinkage, treating an additional diagnosis of urban sprawl as a false positive solely on that basis could penalize a valid diagnosis. Even if an agent reports all five issue families, only those that match the task’s reference annotations count as hits. We acknowledge that this design also leaves unsupported extra diagnoses without a direct penalty.
We plan to add a consistency check to Layer A. If the final report makes incompatible claims about the same indicator in the same area over the same period and leaves the contradiction unresolved, any diagnosis that relies on those claims receives . For example, a report might claim that the population of the same area both increased and decreased between 2015 and 2020.
Matching predictions to references. Experts extract three pieces of reference information from each source study: the diagnosed issue, its location, and the indicators that support the diagnosis. In Valencia, the reference issue is a surface urban heat island, and the reference location is the old town. The reference indicators are summer land surface temperature, Gi* hotspots, low vegetation cover, and impervious surface fraction. Equations (1)–(4) match each diagnosis that the agent explicitly supports in its final report against these references.
- Recognition (A). If the report explicitly diagnoses a surface urban heat island, the target receives A = 1. If it lists the heat island only as a candidate for further investigation, A = 0, and B, C, and the Workflow Score are also zero for that target.
- Localization (B). Only locations linked to the diagnosis are scored. For region tasks, place names or equivalent descriptions are matched to reference areas. B is the fraction of reference areas matched. For city tasks, reported locations are mapped to geographic areas. B is the fraction of the reference area covered by the prediction. If the predicted area exceeds 25% of the city bounding box, B = 0. Within this limit, extra predicted area carries no separate penalty.
- Indicator F1 (C). Indicators used to support the diagnosis are matched to reference indicators by meaning. Equivalent terms count once. For example, if the report gives only summer land surface temperature and impervious surface fraction, it matches two of the four Valencia reference indicators. Thus, C = 2 × 2 / (2 + 4) = 2/3. Both extra and missing indicators lower F1. A proxy with a different meaning can provide supplementary evidence but does not automatically count as a match. Numerical atomic tasks are scored against task-specific reference values and tolerances.
In Equation (5), . Each of the 22 workflow reference annotations corresponds to one target. When several targets share the same input, each answer to that input is scored against each target. We first average the scores within each target and then take the equally weighted mean across all 22 targets. The expert validation in Appendix A.3 covers 56 trajectories and 716 applicable checkpoint judgments.
Expert Validation. Expert involvement is integrated into the benchmark construction. Both the ground-truth references and outcome evaluation criteria were manually developed and reviewed by domain experts. Beyond the checkpoint-level validation described in Appendix A.3, we additionally conducted expert spot-checks of sampled final outcomes to assess the reliability of the reported scores.
Reference
Response to Reviewer ZCWL for W2, Q3 (3/4)
Q3: Could you provide the repetition counts, decoding settings, and instantiated environment-action limits used for each system? Please specify any configured token limits and the exact model/checkpoint where needed to fix the evaluated versions. Are uncertainty estimates for the main comparisons available? For the lightweight LAMBDA adaptation, which original collaboration mechanisms were retained?
A3: System settings. The five agent systems evaluated in the paper share a budget of 25 environment actions per workflow in their latest configurations. Each executed tool or code action counts toward this budget. The Qwen runs use temperature 0.2. The latest configurations set no client-side output-token cap.
- Direct uses two model turns. It selects a batch of actions and then synthesizes the observations.
- Plan-and-Execute uses two model turns. It fixes the plan before execution and synthesizes the results afterward. The executor allows up to two worker retries.
- ReAct selects one action at a time and observes the result before selecting the next, up to the shared 25-action limit. It allows five protocol errors and two finalization attempts.
- MAF-Magentic allows 40 model calls across up to 15 coordination rounds. The higher call limit accommodates the Manager and Reviewer calls in addition to the Analyst’s environment actions.
- LAMBDA allows 27 model calls for up to 25 environment actions, leaving room for final synthesis. As described in Appendix C.5, the original multi-agent system uses an interactive interface and external execution components. DUDA-Bench evaluates all systems in a common execution environment. We therefore followed the sequential tool-execution setup of EvoDS[1]. Our adaptation retains LAMBDA’s iterative analysis loop. The agent selects a function, observes its result, and uses that feedback to choose its next action. A single agent runs this loop in the benchmark.
Repetitions and uncertainty. The original comparisons used one run per configuration. To measure run-to-run variation, we repeated all 22 ReAct workflows twice for each of qwen3.8-max and qwen3.8-27b under the same 25-action budget. Across these two repetitions, the mean Workflow Score was 23.8% for Max and 23.4% for 27B. The sample standard deviations were 3.6 and 2.5 percentage points, respectively. Table R1 also gives the original scores and the three-run means.
Table R1. Workflow outcomes by model and run.
| Model | Run | Recognition (A) ↑ | Localization (B) ↑ | Indicator F1 (C) ↑ | Workflow Score ↑ |
|---|---|---|---|---|---|
| Qwen3.8-Max | Original | 0.727 | 0.274 | 0.281 | 0.277 |
| Repeat 1 | 0.545 | 0.247 | 0.178 | 0.213 | |
| Repeat 2 | 0.636 | 0.308 | 0.220 | 0.264 | |
| Three-run mean | 0.636 | 0.276 | 0.227 | 0.251 | |
| Qwen3.8-27B | Original | 0.432 | 0.125 | 0.141 | 0.133 |
| Repeat 1 | 0.614 | 0.284 | 0.219 | 0.252 | |
| Repeat 2 | 0.591 | 0.219 | 0.213 | 0.216 | |
| Three-run mean | 0.545 | 0.209 | 0.191 | 0.200 |
Reference [1] Z. Yang, F. Liu, Y. Ning, and H. Liu, “EvoDS: Self-Evolving Autonomous Data Science Agent with Skill Learning and Context Management,” in Proc. ACM SIGKDD (KDD), 2026, pp. 6128–6139.
Response to Reviewer ZCWL for W2, Q4 (4/4)
Q4: Figure 2(B1) evaluates routing and tool removal in the no-code setting. What data access remains after both code execution and domain tools are removed? Are independent component ablations with code retained, workflow-linked atomic analyses, or controlled runs with correct intermediate evidence available? These would help assess whether the component findings extend to the full system and distinguish coordination difficulties from failures of the underlying analytical operations. If such analyses are unavailable, please clarify the intended scope of these conclusions.
A4: Independent component ablations with code retained. Figure 2(B1) removes skill routing or domain tools after disabling code execution. With both code execution and domain tools disabled, the agent receives descriptive metadata in the data card and has no way to inspect the underlying data. In the new ablations, we retained code execution and removed either skill routing alone or the analytical and visual tools alone. The full system and both ablations used the same budget of 25 environment actions.
Table R2. Independent ablations with code retained (Qwen3.8-Max).
| Model | Condition | Recognition (A) ↑ | Localization (B) ↑ | Indicator F1 (C) ↑ | Workflow Score ↑ |
|---|---|---|---|---|---|
| Qwen3.8-Max | Full (mean of 2 runs) | 0.591 | 0.278 | 0.199 | 0.238 |
| w/o skill routing (1 run) | 0.591 | 0.283 | 0.208 | 0.245 | |
| w/o tools (1 run) | 0.682 | 0.431 | 0.233 | 0.332 |
With analytical and visual tools removed, the agent could still inspect data and compute indicators in Python. Across the 22 workflows, the trace records 471 code actions in this condition, compared with 302 in each full-system run. The largest score increase was in localization (0.278 to 0.431). These observations suggest that the agent used code to perform more of the analysis itself. With one run for this ablation, the improvement cannot be attributed to tool removal alone.
Workflow-linked atomic analyses. We matched all 22 workflows in the original Qwen3.8-Max ReAct run to their constituent atomic tasks. Each of the 15 workflows that failed the strict success criterion had at least one atomic task the agent solved in isolation. Of their 56 atomic tasks, 26 passed (46.4%) and 30 failed (53.6%). In the Mumbai sprawl case, the agent correctly identified the direction of urban expansion in the atomic task. In the paired workflow, it recognized sprawl and retrieved grid coordinates at step 12 that placed row 1 at the southern edge of the study area. Its final report nevertheless described cells r1c3 and r1c4 as the “NW periphery.” This shows where coordination broke down when the agent translated the coordinates into its final diagnosis.
Response to Reviewer ZCWL for W1, Q1 (1/4)
A1: Thank you for your constructive comments and suggestions. We first compare the observation dates and study area in the source paper with the years and geographic boundary of the benchmark case. We then check the published diagnosis against measurements from the available data within that boundary. We use the Valencia old-town heat-island task to illustrate both checks.
Valencia example. Wei and Sobrino [2] used 17 ECOSTRESS and Landsat images acquired during two summer heatwaves in 2022 and 2023. They compared land surface temperature (LST) across Valencia’s local climate zones and found elevated surface heat in compact built-up areas. Both summers fall within the 2015–2024 benchmark window. We use this finding to define an open workflow task for Valencia old town. The agent receives regional Landsat LST, land cover, and imagery for a fixed bounding box
[-0.40, 39.45, -0.34, 39.50]. It considers all five issue families and reports a diagnosis, affected locations, and supporting evidence. For reference, we have submitted the Valencia case in the supplementary material. We also provide the benchmark implementation and part of the data acquisition code. The complete tasks and dataset will be released upon acceptance.Checking the Valencia reference against the observations. Experts mapped the published finding to three reference components: surface urban heat island as the issue family, Valencia old town as the target area, and summer LST, Gi* hotspots, low vegetation cover, and impervious-surface fraction as diagnostic indicators. We then computed the atomic references on the benchmark data for that same area. The latest summer Landsat raster gives a mean LST of 317.567 K and a 95th percentile of 319.942 K. The hottest cell in the 4 × 4 grid is
(3,1), counting from zero at the northwest corner. Overlaying the high-LST pixels with WorldCover shows that 97.35% of pixels above the 90th temperature percentile fall on non-green land. This checks the link between heat and land cover within the task boundary. In the workflow evaluation, the agent must identify the issue, locate it, and report indicators that match the references. The supplementary material includes the complete Valencia case.Thank you for pointing out the mismatch between the dates of the cited Phoenix study and the benchmark observation period. The study of land-cover change between 1992 and 2001 provided the problem background and analytical basis for the Phoenix task. The ground truth for the urban sprawl diagnosis comes from Fan et al. [1]. Our original manuscript omitted this ground-truth reference. Fan et al. mapped land-cover changes across metropolitan Phoenix using observations from 2001, 2011, and 2021. Their 2021 map falls within the benchmark window of 2015–2024 and documents where agricultural and other land had been converted to developed land. The study therefore supports the diagnosis through 2021.
References
[1] C. Fan, M. J. I. J. Prothan, Y. Zhu, and D. Shi, “Two Decades of Urban Transformation and Heat Dynamics in a Desert Metropolis: Linking Land Cover, Demographics, and Surface Temperature,” Land, vol. 14, no. 11, Art. no. 2141, 2025. https://doi.org/10.3390/land14112141
[2] L. Wei and J. A. Sobrino, “Surface urban heat island analysis based on local climate zones using ECOSTRESS and Landsat data: A case study of Valencia city (Spain),” International Journal of Applied Earth Observation and Geoinformation, vol. 130, Art. no. 103875, 2024. https://doi.org/10.1016/j.jag.2024.103875