- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a new benchmark dataset, called GeBench, which consists of 16,000 synthetically generated question-answer pairs. For each question, it contains one city-scale satellite image, one target-level (building-level) remote sensing image, a structured geographic context features (e.g., the number of nearby points of interest and population-related indicators derived from Google Map APIs and the WorldPop dataset), a short dialogue with a set of QA pairs, and the ground-truth numeric targets which are used only for evaluation. Five building types are considered: schools, hospitals, libraries, houses, and park trails. Two tasks are considered: numeric target prediction (see Table 1) and facility localization. An agentic baseline has been developed to solve this task.
Critical strengths
- The dataset construction is described very clearly.
- The task setup and evaluation procedure are presented clearly.
- The agentic framework is described clearly.
Critical weaknesses
There are several significant weaknesses:
- The most critical drawback is that the tasks presented by GeBench are not difficult enough. Directly predicting the numeric value of a given facility based on its satellite image and geographic context is a classic setup. However, this is usually treated as a well-organized supervised learning task. To evaluate the effectiveness of agentic frameworks on geospatial tasks, the most critical thing is whether it can help to perform geospatial reasoning, geospatial computation, and combine them with common semantic reasoning. Only the localization task is a meaningful geospatial reasoning task. The facility numerical prediction task is a common CV task. If this paper was submitted 2 years ago, I would think it is interesting. Right now, given so many more interesting and challenging geospatial QA benchmarks, I do not think it is a good complement.
- There are no agentic baselines other than the one proposed by the authors. It is hard to access how hard the proposed task is.
- The authors did not do enough literature review and miss many important benchmark works, such as MapEval [1] and CartoMapQA [2]. Moreover, there are some recent agentic LLM works for geospatial reasoning, such as Spatial-Agent [3], Geospatial Agentic Service [4], Collab-rec [5]. These works are not mentioned in the related work section, let alone using them as baselines.
- There are lots of citation errors (\citet instead of \citep)
- Figure 6 shows 5 tasks, but its caption says six tasks. It is confusing.
- The author only uses open-source LLMs. No closed source LLM has been used. I guess the current close-sourced LLMs can already solve many of these tasks.
[1] Dihan, M. L., Hassan, M. T., Parvez, M. T., Hasan, M. H., Alam, M. A., Cheema, M. A., ... & Parvez, M. R. (2024). Mapeval: A map-based evaluation of geo-spatial reasoning in foundation models. ICML 2024.
[2] Ung, H. Q., Habault, G., Nishimura, Y., Niu, H., Legaspi, R., Oya, T., ... & Liu, Y. (2025, November). CartoMapQA: A Fundamental Benchmark Dataset Evaluating Vision-Language Models on Cartographic Map Understanding. In Proceedings of the 33rd ACM International Conference on Advances in Geographic Information Systems (pp. 440-453).
[3] Bao, R., Yang, C., Yu, D., Tang, Z., Mai, G., & Zhao, L. (2026, July). Spatial-Agent: Agentic geo-spatial reasoning with scientific core concepts. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 14896-14911).
[4] Li, Z., Khosravi Kazazi, A., Liu, R., Akinboyewa, T., Ning, H., Huang, X., ... & Simonis, I. (2026). Geospatial Agentic Services: a framework for interoperable geospatial intelligence. Annals of GIS, 1-21.
[5] Banerjee, A., Satish, A., Nur Aisyah, F., Wörndl, W., & Deldjoo, Y. (2026). Collab-rec: An llm-based agentic framework for balancing recommendations in tourism. ACM Transactions on Recommender Systems.
Questions for authors
- How do you design the question template? How do you make the templates diverse enough to cover linguistic variation?
- If the Map context and WorldPop data are pre-collected and stored, how do we expect the agentic framework do? One of the advantages of this dataset, as said by the authors, is "enabling active information acquisition from". If they are provided as data context. This will make the task much easier.
- Why these is no evaluation metric to evaluate the quality of the generated reasoning workflow?