- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces UrbanAgent, a tool-augmented LLM agent for cross-system urban tasks. Addressing fragmented city services and ambiguous user demands, its four-module closed-loop framework clarifies missing information, executes ReAct-style reasoning via MCP-integrated urban tools. The paper also constructs UrbanEval, a 250-query benchmark measuring both task results and execution quality. Evaluated on multiple LLMs, UrbanAgent reaches 71% task success rate with GPT-5-mini, though it suffers higher token overhead and uncertain cross-city generalization.
Critical strengths
- The paper addresses a meaningful problem of coordinating fragmented urban services, enabling agents to handle complex user requests involving multiple systems and constraints.
- UrbanEval evaluates both task success and execution quality, including tool coverage, dependency order, and evidence grounding.
Critical weaknesses
- UrbanEval contains only 250 queries, which is relatively small for evaluating diverse cross-system urban tasks.
- UrbanAgent mainly integrates existing techniques, including intent parsing, ReAct-style reasoning, MCP-based tool invocation, and response verification. The methodological novelty beyond combining these components remains unclear.
- The paper lacks component-level ablation studies to isolate the contributions of cognition, reasoning–execution, tool invocation, and synthesis. Consequently, it is unclear which components drive the reported performance improvements.
- The evaluation relies heavily on a single LLM judge (Qwen3-Max) for task success and other key metrics, without reporting human validation against independent judges. This raises concerns about the reliability of the evaluation results.
Questions for authors
- Could the authors provide component-level ablation studies, such as removing the cognition or synthesis module, to clarify which components contribute most to UrbanAgent's performance improvements?
- Have the authors evaluated the agreement between Qwen3-Max and human annotators or alternative LLM judges to validate the reliability of the reported evaluation results?
- The authors could provide some case studies to help readers better understand the workflow of UrbanAgent.
Suggestion: I have previously seen the name "UrbanAgent" used in 'Multi-Agent Collaborative Reasoning with Tool-Augmented Evidence for Urban Region Profiling' (KDD 2026). I suggest the authors consider renaming their framework to avoid confusion with this existing work.