acceptodds
Withdrawn from ICLR 2027

UrbanAgent: A Tool-Augmented Agent for Cross-System Urban Tasks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern cities rely on an increasing number of digital services to operate, but residents’ daily needs are still difficult to meet. Services are fragmented and have little interoperability, placing a heavy operational burden on users. Existing digital platforms, urban foundation models, and intelligent assistants each address only isolated aspects of an urban task. But they struggle to reliably convert complex natural-language requests into executable cross-system workflows. We propose UrbanAgent, a tool-augmented agent framework for cross-system urban tasks. It couples the cognitive and reasoning capabilities of a large language model with a toolset supporting code execution, API calls, and Model Context Protocol. Through one adaptive closed loop, it clarifies missing information before acting, grounds tool use in live observations, and aligns the final response with observed evidence and task constraints. To address the evaluation gap, we introduce UrbanEval, a benchmark specifically designed for cross-system urban request. Unlike prior benchmarks that assess either general tool use or urban knowledge and reasoning, UrbanEval evaluates both task results and execution quality, including required tool coverage, dependency validity, and evidence traceability. Experimental results indicate that UrbanAgent reaches a 71% task success rate, 10points above the strongest baseline. This lead holds across GPT-5-mini, Gemini-2.5-flash, DeepSeek-V4-flash, and Qwen3-235B-A22B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tool-Augmented Agents, Large Language Models, Urban Intelligence, Multi-Tool Orchestration, Agent Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_J61jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces UrbanAgent, a tool-augmented LLM agent for cross-system urban tasks. Addressing fragmented city services and ambiguous user demands, its four-module closed-loop framework clarifies missing information, executes ReAct-style reasoning via MCP-integrated urban tools. The paper also constructs UrbanEval, a 250-query benchmark measuring both task results and execution quality. Evaluated on multiple LLMs, UrbanAgent reaches 71% task success rate with GPT-5-mini, though it suffers higher token overhead and uncertain cross-city generalization.

Critical strengths
  1. The paper addresses a meaningful problem of coordinating fragmented urban services, enabling agents to handle complex user requests involving multiple systems and constraints.
  2. UrbanEval evaluates both task success and execution quality, including tool coverage, dependency order, and evidence grounding.
Critical weaknesses
  1. UrbanEval contains only 250 queries, which is relatively small for evaluating diverse cross-system urban tasks.
  2. UrbanAgent mainly integrates existing techniques, including intent parsing, ReAct-style reasoning, MCP-based tool invocation, and response verification. The methodological novelty beyond combining these components remains unclear.
  3. The paper lacks component-level ablation studies to isolate the contributions of cognition, reasoning–execution, tool invocation, and synthesis. Consequently, it is unclear which components drive the reported performance improvements.
  4. The evaluation relies heavily on a single LLM judge (Qwen3-Max) for task success and other key metrics, without reporting human validation against independent judges. This raises concerns about the reliability of the evaluation results.
Questions for authors
  1. Could the authors provide component-level ablation studies, such as removing the cognition or synthesis module, to clarify which components contribute most to UrbanAgent's performance improvements?
  2. Have the authors evaluated the agreement between Qwen3-Max and human annotators or alternative LLM judges to validate the reliability of the reported evaluation results?
  3. The authors could provide some case studies to help readers better understand the workflow of UrbanAgent.

Suggestion: I have previously seen the name "UrbanAgent" used in 'Multi-Agent Collaborative Reasoning with Tool-Augmented Evidence for Urban Region Profiling' (KDD 2026). I suggest the authors consider renaming their framework to avoid confusion with this existing work.

Reviewer_DnVuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies natural-language urban requests that require information or actions from several separate services. UrbanAgent parses the request into an objective, entities, hard and soft constraints, and missing information. A ReAct-style controller then calls registered tools, including MCP services, and a synthesis stage checks the proposed response against the observations and stated constraints. The paper also introduces UrbanEval: 250 requests divided evenly among factual retrieval, complex retrieval, sequential dependency, conditional/combinatorial decisions, and ambiguous or incomplete intent. Its metrics cover task success, clarification, tool coverage, dependency order, proactive verification, and token use. The method ranks first in task success under all four tested base models, although its margin varies.

Critical strengths
  1. The evaluation asks a consequential question about agents. UrbanEval checks whether a request was actually completed with supporting observations, whether the required tool capabilities were used, and whether dependent calls occurred in the right order. This design can expose an agent that gives a plausible final answer while skipping a required verification step, which answer-only benchmarks often miss.
  2. The experiments show improvements gained by the designed agent. The GPT-5-mini gain remains 10.5 percentage points when the 50 clarification requests are excluded. UrbanAgent also ranks first in task success for Gemini-2.5-flash, DeepSeek-V4-flash, and Qwen3-235B-A22B. The paper reports where the gains concentrate and where they do not, including a category in which AutoGen is slightly better.
Critical weaknesses
  1. The difference between an urban agent and a deep research agent is not clear. UrbanAgent’s clarification, iterative tool use, evidence gathering, and final verification substantially overlap with deep-research-style agents, which consists a much more general workflow. This also makes novelty marginal since what this task is doing is basically what a deep research agents is doing. Also, the agent itself largely combines familiar parsing, ReAct-style execution, and answer checking.
  2. The outcome is not objective. Task success and two other metrics are assigned by Qwen3-Max using fixed rubrics and masked system identities. Masking and archived verdicts improve auditability, but the paper does not report a blinded human adjudication study, agreement statistics, or an error analysis on cases where the methods disagree. This matters because a 21-task difference on the executable subset drives the main claim; a judge that systematically rewards UrbanAgent's more detailed traces could overstate that difference. The deterministic trace metrics cannot substitute for validating the task-success verdicts.
  3. The system gain is based on large token consumption. The paper attributes the advantage to augmentation and completion, yet it does not remove the parsing, controller, or synthesis components individually. On GPT-5-mini, tool coverage and invocation order are already close to baseline values, while UrbanAgent uses 118.3k tokens per successful task for the strongest competing systems. Thus the results establish improvement under a common maximum budget, but not how much is due to the proposed structure rather than additional inference and verification.
Questions for authors
  1. How would a deep-research-style agent perform on the 200 executable UrbanEval requests when given the same urban tools, base model, and call/token budget? Which UrbanAgent component provides a capability that this baseline lacks?
  2. What are the success rates and token costs after removing clarification/parsing, the adaptive controller, and the final synthesis check one at a time? Which removal accounts for the most of the 21 additional executable-task successes?
  3. At approximately the same tokens per request or per success, does UrbanAgent still outperform native tool calling or AutoGen? A limited comparison on the 200 executable requests would directly test the practical trade-off without requiring a new benchmark.
Reviewer_YLN8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes UrbanAgent, a tool-augmented large-language-model agent for urban requests that span multiple services. Its pipeline parses a request into intent, entities, hard and soft constraints, and missing information; uses a ReAct-style controller to call code, APIs, and MCP tools; and applies a synthesis stage intended to align the final answer with collected observations and constraints. The paper also introduces UrbanEval, a 250-query benchmark with 50 examples in each of five categories: factual retrieval, independent complex retrieval, sequential dependency, conditional or combinatorial decision making, and ambiguous or incomplete intent. The evaluation combines result metrics, process metrics derived from traces, and token-efficiency metrics. Qwen3-Max is used to judge task success, missing-information handling, and proactive augmentation. In the GPT-5-mini setting, UrbanAgent reports 71 percent overall task success versus 61 percent for the strongest baselines, and 64.0 percent versus 53.5 percent on the 200 executable queries. UrbanAgent ranks first in task success across four base models, but uses substantially more tokens; the reported margin falls to two points with DeepSeek-V4-flash.

Critical strengths
  1. A relevant problem and a better evaluation unit. Cross-service urban requests are a reasonable and under-evaluated agent setting. UrbanEval's per-query checklists and trace-aware process metrics attempt to distinguish a plausible final answer from a workflow that used the required capabilities in a valid order and grounded its claims in observations. This is more diagnostic than evaluating only the final text.
  2. Broad controlled coverage and transparent cost reporting. The experiments cover four orchestration baselines and four base models under a common tool surface and budget within each model block. The paper reports both the consistent rank of UrbanAgent and the substantial token cost, and it breaks results down by query type and failure class. This makes the empirical tradeoff more visible than a single aggregate score would.
Critical weaknesses
  1. The experiment does not identify what causes UrbanAgent's gain. UrbanAgent bundles several modules and hand-designed execution policies while using roughly three times as many tokens per successful GPT-5-mini task as the strongest baselines. Without a complete component ablation, prompt- and compute-matched controls, or repeated runs, the gain cannot be attributed to the proposed architecture rather than specialized instructions, extra inference, or stochastic variation.
  2. The LLM-judge evaluation is not adequately validated. TSR, MIR, and VPAR depend on one Qwen3-Max judge, but the paper reports neither blinded human agreement nor sensitivity to an independent judge. The reported ranking may therefore be evaluator-specific.
  3. The released provenance and score accounting contradict the reproducibility claim. The exact runtime and prompt hashes are unavailable, eight result files were overwritten, only 242 of 250 metric rows match the current archive, and the released prompt was not evaluated. Five null UrbanAgent verdicts also conflict with the 250-query denominator and the documented exclusion of null judgments, so the headline results cannot be independently reconstructed.
Questions for authors
  1. Can the authors provide the immutable source snapshot, complete prompts, environment, tool revisions, and original artifacts for every reported system, and reconcile the eight overwritten runs? Does the current release reproduce the headline table; if not, which claims should change?
  2. What are the results of a complete component ablation under the same endpoint, judge, queries, tool snapshot, and budget, with prompt- and compute-matched ReAct controls and repeated runs or paired confidence intervals? This would show whether the gain extends beyond specialized prompting and additional inference.
  3. How well do Qwen3-Max decisions agree with blinded human annotators and an independent judge on task success and VPAR? Please report the sample, rubric, inter-annotator agreement, and any change in system ranking across judges.
  4. What is the benchmark and ground-truth freeze timeline, and how were null or pending items and ambiguous requests handled in TSR? Please reconcile five null UrbanAgent decisions with the 250-query denominator and the statement that all 72 non-success executable cases are failures, then report explicit numerators and denominators.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.