acceptodds
Under review as a conference paper at ICLR 2027

GEBench: Benchmarking Tool-Augmented Agents for Geo-Economic Reasoning

Abstract

abstract Geo-economic decision-making, like public-facility assessment and allocation, requires not only interpreting multimodal geospatial information but also actively discovering what data is missing and which geographic tools can supply it. While recent geospatial foundation models and multimodal large language models have improved perception and geo-grounding, the existing benchmarks predominantly focus on single-modal inputs and tasks related to remote sensing, leaving tool-driven economic and demographic knowledge discovery underexplored. This paper therefore introduces the Geo-Economic Benchmark (GEBench), a large-scale multimodal benchmark for predictive geo-economic reasoning grounded in remote sensing imagery, map-derived place context, and demographic priors. To the best of our knowledge, this is the first work to benchmark agentic, tool-interactive methods for solving geo-economic problems. GEBench formulates geo-economic tasks as question-answer pairs with multi-modal input and supports multi-round interaction. It comprises over 16,000 question-answer pairs across five building categories with their corresponding geo-economic prediction targets and spanning more than 6 countries. Building on GEBench, we develop an agentic reasoning pipeline in which an exploration agent plans feasible API queries and iteratively integrates results from mapping and demographic tools before downstream analysis and forecasting, enabling active information acquisition from Geo-Economic data. We systematically evaluate tasks including numerical prediction and localization, demonstrating that our agentic reasoning framework consistently outperforms direct model querying across diverse geo-economic scenarios.

Primary area:
datasets and benchmarks
Keywords:
Geo-economic task, Agentic evaluation benchmark, Multimodal agentic reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uBRYOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a new benchmark dataset, called GeBench, which consists of 16,000 synthetically generated question-answer pairs. For each question, it contains one city-scale satellite image, one target-level (building-level) remote sensing image, a structured geographic context features (e.g., the number of nearby points of interest and population-related indicators derived from Google Map APIs and the WorldPop dataset), a short dialogue with a set of QA pairs, and the ground-truth numeric targets which are used only for evaluation. Five building types are considered: schools, hospitals, libraries, houses, and park trails. Two tasks are considered: numeric target prediction (see Table 1) and facility localization. An agentic baseline has been developed to solve this task.

Critical strengths
  1. The dataset construction is described very clearly.
  2. The task setup and evaluation procedure are presented clearly.
  3. The agentic framework is described clearly.
Critical weaknesses

There are several significant weaknesses:

  1. The most critical drawback is that the tasks presented by GeBench are not difficult enough. Directly predicting the numeric value of a given facility based on its satellite image and geographic context is a classic setup. However, this is usually treated as a well-organized supervised learning task. To evaluate the effectiveness of agentic frameworks on geospatial tasks, the most critical thing is whether it can help to perform geospatial reasoning, geospatial computation, and combine them with common semantic reasoning. Only the localization task is a meaningful geospatial reasoning task. The facility numerical prediction task is a common CV task. If this paper was submitted 2 years ago, I would think it is interesting. Right now, given so many more interesting and challenging geospatial QA benchmarks, I do not think it is a good complement.
  2. There are no agentic baselines other than the one proposed by the authors. It is hard to access how hard the proposed task is.
  3. The authors did not do enough literature review and miss many important benchmark works, such as MapEval [1] and CartoMapQA [2]. Moreover, there are some recent agentic LLM works for geospatial reasoning, such as Spatial-Agent [3], Geospatial Agentic Service [4], Collab-rec [5]. These works are not mentioned in the related work section, let alone using them as baselines.
  4. There are lots of citation errors (\citet instead of \citep)
  5. Figure 6 shows 5 tasks, but its caption says six tasks. It is confusing.
  6. The author only uses open-source LLMs. No closed source LLM has been used. I guess the current close-sourced LLMs can already solve many of these tasks.

[1] Dihan, M. L., Hassan, M. T., Parvez, M. T., Hasan, M. H., Alam, M. A., Cheema, M. A., ... & Parvez, M. R. (2024). Mapeval: A map-based evaluation of geo-spatial reasoning in foundation models. ICML 2024.

[2] Ung, H. Q., Habault, G., Nishimura, Y., Niu, H., Legaspi, R., Oya, T., ... & Liu, Y. (2025, November). CartoMapQA: A Fundamental Benchmark Dataset Evaluating Vision-Language Models on Cartographic Map Understanding. In Proceedings of the 33rd ACM International Conference on Advances in Geographic Information Systems (pp. 440-453).

[3] Bao, R., Yang, C., Yu, D., Tang, Z., Mai, G., & Zhao, L. (2026, July). Spatial-Agent: Agentic geo-spatial reasoning with scientific core concepts. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 14896-14911).

[4] Li, Z., Khosravi Kazazi, A., Liu, R., Akinboyewa, T., Ning, H., Huang, X., ... & Simonis, I. (2026). Geospatial Agentic Services: a framework for interoperable geospatial intelligence. Annals of GIS, 1-21.

[5] Banerjee, A., Satish, A., Nur Aisyah, F., Wörndl, W., & Deldjoo, Y. (2026). Collab-rec: An llm-based agentic framework for balancing recommendations in tourism. ACM Transactions on Recommender Systems.

Questions for authors
  1. How do you design the question template? How do you make the templates diverse enough to cover linguistic variation?
  2. If the Map context and WorldPop data are pre-collected and stored, how do we expect the agentic framework do? One of the advantages of this dataset, as said by the authors, is "enabling active information acquisition from". If they are provided as data context. This will make the task much easier.
  3. Why these is no evaluation metric to evaluate the quality of the generated reasoning workflow?
Reviewer_mkh2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a multimodal benchmark with over 16,000 QA pairs for predicting geo-economic variables across five building types. It also proposes a multi-round agentic workflow that integrates multi-source geospatial and demographic data before producing the final answer.

Critical strengths

The paper proposes the GEBench, a benchmark with three types of task measurement. The first is numeric prediction: predicting geo-economic variables, such as the number of students and teachers or revenue margin and net income, across five building types. The second is localization: predicting a facility’s pixel coordinates in a satellite image. The third is image causal removal: the benchmark provides edited satellite images in which the target facility is masked at the instance level and removed, which allows counterfactual reasoning and causal analysis.

Critical weaknesses

The completeness of the paper needs substantial improvement.

  1. Evaluation side: Only six general-purpose MLLMs are evaluated. It's recommended to add three groups of models: representative remote sensing/geospatial MLLMs, closed-source frontier models or more capable open-source models, and specialized prediction models. This would make the benchmark more informative and allow for more diagnostics. Such an evaluation portfolio would also follow common practice in MLLM benchmarking papers.

By adding specialized models (e.g., an XGBoost regressor and a CNN baseline), readers would gain a sense of how existing tools perform on this task, providing stronger motivation for introducing MLLMs into geo-economic tasks. By including a broader range of frontier general-purpose MLLMs, readers could gain a more comprehensive understanding of how far current models are from satisfactory performance. Finally, adding remote sensing MLLMs (agents) would help clarify their capabilities and limitations in geo-economic reasoning, as well as the distinction between general geospatial understanding and geo-economic reasoning. Together, these comparisons would highlight the distinctive challenges of the proposed tasks and strengthen the paper’s contribution.

In addition, although Section 3.3 (Evaluation Protocol and Metrics) introduces image causal removal for counterfactual reasoning, no task or metric is defined for it and no experiment uses it.

  1. Method side: The effectiveness of the proposed agentic workflow needs more justification. Currently, it is validated only on the numeric prediction task (agentic vs. direct). It would be helpful to have comparisons on more diverse task types and ablations of the workflow’s modules (e.g., sub-agents or tools). A generalizable framework with clear module attribution would further strengthen the technical contribution.
Questions for authors
  1. In Table 3, direct answering generally outperforms the agentic workflow on house price prediction. Could the paper discuss why? In addition, the gaps between some agentic and direct results are quite small. Reporting results over multiple runs with standard deviations would help clarify model behavior and stability.

  2. Table 4 shows that the library and trail subsets are built from only 18 and 30 unique locations, respectively, yet yield 3,000 and 1,000 dialogues. What distinguishes dialogues generated from the same site, and is that information part of the model input? More broadly, could the authors report diversity statistics for the 16,000 QA pairs, such as the number of templates or target values per location?

Reviewer_chrYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces GEBench for evaluating tool-augmented agents on geo-economic analysis. The benchmark combines satellite imagery, map-derived context, and population features with numerical targets for schools, hospitals, libraries, residential properties, and trails, yielding approximately 16,000 dialogue records. The authors also propose an agentic workflow comprising exploration, image analysis, context analysis, forecasting, evaluation, and reporting, with cached tool responses supporting repeatable interactive evaluation. Experiments compare direct querying and agentic inference across six open VLM backbones. The results suggest that the multi-stage workflow can reduce prediction error for some tasks and models, although the improvements are not consistent.

Critical strengths

The task direction is practically relevant. Connecting multimodal geographic information to structured numerical targets such as facility capacity, visitor counts, and property values moves beyond image description and geographic knowledge QA toward practical analytical needs. The categories also offer opportunities to study how visual and contextual features support socioeconomic estimation.

Critical weaknesses

1、Several inconsistencies in the core results prevent reliable interpretation of the evaluation. Section 3.3 specifies MAE for hospitals, whereas Table 3 labels all columns as MAPE. Section 4.2 states that Table 3 reports prediction error and failure rate, but no failure-rate column is provided. Figure 6 shows five category axes, while the accompanying text interprets it as a six-dimensional comparison including localization and parsing performance. The claim in Section 4.3 that agentic inference consistently outperforms direct querying is also too strong: among the 28 comparable pairs in Table 3, eight worsen and one ties, with deterioration for five backbones in the House category. These issues concern the interpretation of the main results rather than presentation alone. The metrics, treatment of failed predictions, and conclusions need to be reconciled.

2、The experiments do not sufficiently establish the active information-discovery capability emphasized by the benchmark and method. Matching the available evidence is a strength, but agentic inference simultaneously introduces additional model calls, role decomposition, and structured validation. Its gains therefore cannot be attributed specifically to active tool selection. A fixed tool-acquisition workflow or a budget-matched single-agent baseline is missing, and tool-selection quality is not sufficiently quantified. The broader claims concerning site selection and causal analysis also exceed the demonstrated evidence: observed facility attributes support prediction tasks, but do not automatically provide labels for optimal placement or counterfactual outcomes after facility removal. The paper should distinguish evaluated attribute estimation and visual localization from prospective applications in planning and causal inference.

Questions for authors

1、Does the Hospital column in Table 3 report MAE or MAPE? Please specify the units, percentage scaling, error clipping, treatment of invalid predictions, and aggregation denominator for each column. Please also report the valid-output rate for each model and inference mode. 2、What metrics are actually plotted in Figure 6, and why do the six dimensions described in the text differ from the five categories shown? Please reconcile the interpretation and discuss the widespread deterioration in the House category in Table 3. 3、With matched evidence and comparable inference budgets, does active tool selection outperform a workflow that retrieves all available evidence in a fixed order? Which results separate the benefit of exploration from those of additional reasoning and output validation? 4、Does the dataset cover three countries or more than six? How are multiple dialogues derived from the same location aggregated and accounted for when estimating uncertainty? For site-selection or image-removal tasks, how are reference answers established and validated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.