acceptodds
Under review as a conference paper at ICLR 2027

From Map to Street: A Benchmark for Multi-Scale Geospatial Reasoning in Instruction-Driven Geographic Localization

14 views

Abstract

Instruction-driven geographic localization is a fundamental geospatial-reasoning task in which a model must fuse multiple sources of evidence to locate a real-world place described by natural-language instructions. Despite its practical importance, existing benchmarks capture only a narrow, static version of this task: they provide fixed inputs, allow little active exploration of geospatial space, and evaluate models primarily by final localization accuracy. To address this limitation, we present **IDOL**, a comprehensive benchmark for **I**nstruction-**D**riven **O**pen-world geographic **L**ocalization, which requires an agent to locate a place that a colloquial instruction describes by fusing multi-modal evidence across map, satellite, and street-level views. The benchmark covers eight cities and seven languages, and defines two task families: pinpointing a unique place from visual cues, and finding any place satisfying an open-ended predicate. For each family, we design a diverse suite of metrics that assess the reasoning process rather than the final answer alone. Experimental results show that while powerful models such as GPT-5.6-Sol and Gemini-3.1-Pro demonstrate great potential, they remain limited in fine-grained visual understanding and spatial grounding; moreover, their performance varies across cities and languages, exposing the imbalance in current models' reasoning capabilities. Our benchmark-generation and evaluation frameworks can be found at https://anonymous.4open.science/r/IDOLBenchmark/.

Primary area:
datasets and benchmarks
Keywords:
Geospatial reasoning, Geographic localization, Multimodal LLM, Visual grounding, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qXUwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces IDOL, a benchmark for geographic localization from natural language instructions within a specified city. Agents access place search, satellite imagery, and street imagery to identify either a particular described location or locations satisfying visually verifiable conditions. The benchmark contains 7,126 query instances across eight cities and seven languages, derived from 1,018 original queries. Its main contributions are the curated dataset, a common framework for agent interaction with geospatial tools, and diagnostic evaluation protocols covering localization accuracy, candidate retrieval, and visual evidence supporting returned locations. Validation includes experiments with multiple multimodal language models, tool ablations for Type 1, comparisons across cities and languages, and human analysis of failure cases. For Type 2, the authors assess three automated judges against human annotations of 600 observed places. The results reveal substantial gaps between retrieving a correct candidate and selecting the final location, demonstrate benefits from visual verification, and highlight differences in how consistently models gather evidence to support their answers.

Critical strengths

Practically relevant task formulation and a useful benchmark resource. IDOL evaluates whether agents can resolve informal location requests within a known city, covering both specific targets and multiple valid locations satisfying visual conditions. Its 1,018 original queries span eight cities, undergo human review, and are provided in seven language versions. This combination of practical tasks, geographic coverage, and reviewed data provides a valuable testbed for studying how multimodal agents connect user instructions to real locations through geospatial tools. The benchmark’s potential usefulness to subsequent research is a substantive strength supporting acceptance. Then, the evaluation distinguishes candidate retrieval from final localization, making important performance gaps visible. For example, Gemini 3.5 Flash achieves 71.1% oracle success within 50 m but only 51.6% final success, revealing substantial room for improvement after candidate retrieval. Type 2 additionally tracks whether returned locations have supporting visual observations and satisfy the requested conditions. Tool ablations further show consistent accuracy losses when StreetView is removed across the six evaluated models. These analyses help identify concrete weaknesses in search, selection, and verification, substantially increasing the paper’s value as an evaluation contribution.

Critical weaknesses

The central motivation for multi-scale geospatial reasoning is insufficiently substantiated. The paper motivates IDOL through the need to fuse evidence across map, satellite, and street-level views, but does not establish how frequently its queries actually require such integration. The agent is given the city and can retrieve candidate names and coordinates, yet Table 7 omits a SearchPlaces-only baseline, leaving the additional difficulty introduced by visual constraints insufficiently characterized. The existing ablations support the usefulness of street-level verification, while the benefits of satellite imagery vary across models and metrics; they do not identify which queries require complementary evidence from different scales. Moreover, Type 2 success requires image observation by definition, so penalizing answers without visual observations cannot independently establish the necessity of visual verification. Without clearer controls distinguishing semantic retrieval, visual matching, and cross-scale reasoning, it remains uncertain whether IDOL specifically measures the capability that motivates its design. This weakens the justification for the benchmark’s central contribution.

Insufficient validation of the Type 2 evaluation and its interpretation. Requiring visual evidence is a reasonable benchmark objective, but the Type 2 success metric combines evidence acquisition with answer validity, leaving unsupported answers indistinguishable from incorrect ones in the headline score (Section 4.2). More critically, Section 5.6 interprets unsuccessful post-hoc visual verification as hallucination, although the judge explicitly permits an “indeterminate” outcome whose treatment is not clarified (Appendix A.8.4). The comparison of three judges and the 600-place human audit are valuable safeguards, but the primary judge achieves 78.7% human agreement and κ = 0.526 (Table 5), and the paper does not report human-validated query-level success rates or uncertainty estimates for the resulting comparisons. Consequently, the evidence does not sufficiently establish how much of the reported failure reflects incorrect locations, missing visual support, or judge error. Because reliable measurement is central to a benchmark contribution, this uncertainty materially weakens my confidence in the Type 2 comparisons and the conclusions about model hallucination.

Questions for authors
  1. Clarification of unsuccessful verification and hallucination. How are the judge’s fails and indeterminate verdicts treated in the subsequent verification analysis in Section 5.6 and Table 6? Could you report their separate frequencies and clarify whether the stated hallucination rate includes answers that cannot be confirmed because the imagery is insufficient, rather than because the location demonstrably violates the query?

  2. Additional evidence on the motivating capability. For Type 1, how well does an agent using only SearchPlaces perform? Across both task families, how does the adaptive agent compare with a fixed pipeline that first searches and then verifies, using the same backbone, available tools, and comparable call budgets? Do these comparisons support the claim that IDOL tests adaptive integration across scales beyond candidate retrieval and visual matching?

  3. Additional validation of the Type 2 comparisons. On a representative subset of queries with complete answer sets, do human judgments of the same retrieved evidence reproduce the main differences in Type 2 Succ and the resulting model rankings? Could you provide uncertainty estimates for these comparisons, given the primary judge’s 78.7% agreement with human annotations in Table 5?

  4. Clarification of consistency between the user query and the judging predicate. The agent receives a colloquial query, whereas the judge additionally receives a precise visual predicate, as described in Appendices A.8.1 and A.8.4. Did the human review explicitly verify that every condition in the predicate is expressed in, or unambiguously implied by, the query visible to the agent? How were cases admitting multiple reasonable interpretations handled?

  5. Additional validation of sensitivity to alternative correct locations. Using the failure audit in Section 5.5, could you quantify how accepting the alternative matches confirmed by human reviewers would affect Type 1 success rates and the main model comparisons? What evidence indicates whether residual ground truth ambiguity materially affects the conclusions, particularly for models with similar scores?

  6. Clarification of the conclusions about performance across languages. What checks established that translations into the other six languages preserve the visual and spatial clues, as well as the ambiguity, of the original Chinese queries? Given that the agent prompts require English search keywords, what evidence distinguishes the language effects reported in Section 5.4 from translation errors or differences in generating English retrieval queries?

Minor:The text in several figures, particularly Figures 1, is difficult to read at normal viewing size.

I am open to revising my score if the authors can provide satisfactory clarifications that address the concerns raised above.

2 comments

Thanks for the detailed response. My main concerns have been addressed.

After the ac address the potential ethical concerns, I will change the score accrodingly.

AuthorsOpenReview ↗

Clarification Regarding Potential Ethical Concerns

Dear Reviewer qXUw,

Thank you for your updated assessment. We are glad to hear that your main concerns have been addressed.

Regarding the potential ethical concerns mentioned in your comment, we currently cannot see any corresponding comment or specific concern in the discussion thread. The visible review also indicates “Flag For Ethics Review: No.” Therefore, we are unsure what issue may still require clarification or action from us.

If there is anything the authors should address, please let us know. We would be happy to provide any necessary clarification promptly.

Thank you again for your time and consideration.

Reviewer_Vp1cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a new benchmark for geo-localization, IDOL, based on text input. The agent receives a text query and tries to find the location coordinates using tools such as 1) keyword-based place search, 2) a satellite tile, and 3) Street View. Geo-localization is a popular research problem, especially with the rise of games like GeoGuessr and related research, and the authors address the opposite problem (where the input is text, not an image). Two task families are defined: 1) Type 1, with one unique target, and 2) Type 2, an open-ended predicate, e.g., "an outdoor pool". The dataset covers 8 cities and 7 languages, including 823 Type 1 and 195 Type 2 queries per language. Queries were translated from the master language using an LLM. 12 LLMs were evaluated across a diverse set of metrics. The results show that even strong language models remain limited and don't perform ideally on the benchmark.

The overall benchmark is well-motivated and includes extensive evaluations. However, several methodological issues should be addressed. The most concerning questions are: a) harness and memory handling, and b) question quality.

Critical strengths
  1. Clear definition of the actual task and an obvious real gap. The authors try to solve the localization problem in the new setup by providing a text query and returning coordinates using agents with tool calls. The benchmark is interesting and has potential to be useful.
  2. Extensive experimental setup, including a diverse set of MLLMs covering a broad range of metrics.
  3. Judge reliability is well measured: 600 human labels, ranked by 3 VLLM judges.
  4. Comprehensive ablation on tool calls in A1.
  5. Easy-to-follow text and code.
Critical weaknesses
  1. It's unclear to me how the model compares candidates without any memory. As written, the model receives only images from the last call at each turn. It doesn't have memory and can't recall previous images. Therefore, an obvious drawback of the method is that the model looks at candidate A, then looks at candidate B, and has no idea what candidate A looked like. It didn't take any notes or cache the data. At the same time, the paper states, “compare how well multiple candidates match the query,” but in reality it doesn't work. The “retrieval vs verification” finding blames the model, but the model couldn't remember the correct candidate. Part of the failure is the harness, not the model.
  2. I'm not sure that such Type 2 questions are valid: "Is there any large public art nearby that can be viewed outdoors?" What does nearby mean? You have only the city name in the context. The query should either drop the word "nearby" to return all suitable locations in the city or include the query's location in the request.
  3. All queries are written in Chinese and then translated into other languages. During this translation, some toponyms were translated directly, and some unique place information was lost. For example, sample_data//Tokyo_Type1_.json, "Unicorn Gundam". The Korean query refers to the Seoul city district (due to a wrong translation), which invalidates part of the dataset. All the translations should be thoroughly verified.
  4. The actual dataset size is much smaller. 5761 Type 1 queries are actually 823 queries × 7 translations (by the LLM, which is sometimes imprecise). Please report the original numbers of unique queries and locations. Several queries also belong to the same place. Please report the number of unique places.
  5. Why does Gemini-3.5-Flash have better MDErr but also lower S@25, S@50, S@100, and S@500 than Gemini 3.1 Pro? Was MDErr computed only over answered queries? Do you report the answer rate somewhere?

Minors:

  • Table 1: 8/7 is highlighted in black, but 8 is obviously less than Global.
  • Can you share some design choices? Why do you use different temperatures for different models
  • ChatGPT has the lowest ObsRat in Table 4. Are images correctly passed to GPT models?
Questions for authors

Authors should consider verifying or better explaining the framework and resolving issues with multi-turn memory retention; add answer rates; tone down the language and dataset-size claims accordingly, or guarantee the full dataset's quality; share design choices and explain related concerns.

Reviewer_RSV2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces IDOL, a benchmark spanning eight cities and seven languages for locating real places from colloquial descriptions without explicit place names or precise locations. Agents use search, satellite imagery, and street views to identify either a unique location or any place satisfying an open-ended visual condition. The best model achieves 50.9% accuracy within 25 meters and 64.6% within 100 meters, with a substantial gap between candidate retrieval and final selection. Models also sometimes answer without visual verification, highlighting limitations in evidence gathering and visually grounded geographic reasoning.

Critical strengths
  1. The main idea is to recast geolocalization as instruction-driven, active, multi-scale, open-world evidence search: map retrieval, a satellite coarse filter, a street-level check, then a location.
  2. The task is close to a real need: a vague visual description and active gathering of evidence at several scales, and the tools fit the task. POI search narrows candidates, satellite view checks layout, and street view checks color, material, and local detail.
  3. Every generated query is checked by hand. About 70 failures per model are blind-reviewed and broken down by error source. Judges from three vendors, with a rank comparison, are more stable than a single LLM judge.
Critical weaknesses
  1. The benchmark depends on the live Google Maps Platform. Search results, street-view coverage, and satellite imagery change over time, so the same trajectory is hard to repeat.
  2. There are only 1,018 independent questions. Seven languages are translations of the same queries.
  3. “Unique within the city” cannot be proved by enumeration. Review only checks retrieved candidates. The failure analysis itself finds 3.6–6.3% incomplete ground truth.
Questions for authors
  1. Query generation and filtering use GPT-5.5, which can imprint its descriptive style and place categories. Tested models may also already know famous landmarks. Can this bias be tested and eliminated?
  2. Open-world tools leak place names, addresses, and popular POI metadata. Some tasks can be solved by text matching rather than by fusing satellite and street view. How to test the real ability?
  3. Should have native speakers write queries independently in local scenes, rather than read translation performance as cross-cultural generalization.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary and Key Revisions

Dear Members of the Program Committee,

To support an efficient and independent assessment of our revised submission, we summarize below the paper’s value and the main improvements made in response to each reviewer.

Overall contribution and value

IDOL introduces an instruction-driven, open-world geographic localization benchmark in which agents must actively search for and visually verify real-world locations from colloquial descriptions. It contains 1,018 human-validated source queries, expanded into 7,126 multilingual instances across eight cities and seven languages, and covers both unique-location and open-ended visual-condition tasks. Its evaluation separates candidate retrieval, final selection, evidence acquisition, and visual verification, thereby exposing failure modes that conventional localization accuracy alone cannot reveal.

All three reviewers recognized the practical relevance of the task, the usefulness of the benchmark, and the breadth of the evaluation. Their main concerns focused on empirical validation, data accounting, and the precision of several claims. We addressed these concerns through the following new experiments and revisions.

Response to Reviewer RSV2

  • Separating parametric knowledge, textual retrieval, and visual reasoning.

We added NoTools and SearchOnly ablations. SearchOnly substantially improves over NoTools, but the full framework consistently outperforms the SearchOnly setting. This demonstrates that IDOL cannot be reduced to landmark memorization or textual POI matching and that visual verification provides measurable additional value.

  • Testing sensitivity to incomplete ground truth.

We used the human failure audit to identify alternative locations confirmed as correct. Accepting these alternatives raises S@100 by only 1.6–3.0 percentage points and leaves the model ranking unchanged. Thus, residual open-world ambiguity may slightly underestimate absolute accuracy but does not materially affect the paper’s comparative conclusions.

  • Systematically validating multilingual data.

We audited all translated query pairs using three independent judges. Majority voting classified 96.1% as Pass, 3.0% as Minor Difference, and only 0.4% as Error. Every identified error was manually reviewed and corrected.

Response to Reviewer Vp1c

  • Directly testing the visual-memory concern.

We evaluated four additional memory strategies on a controlled 150-query subset: retaining all images, retaining the five most recent images, storing VLM-generated image summaries, and allowing summary-based image recall. The results show that memory effects are model-dependent and that no strategy consistently improves threshold success.

  • Making dataset size and metrics fully transparent.

We now separately report 1,018 source queries, 7,126 language-specific instances, 823 Type 1 queries, 195 Type 2 queries, and 367 distinct Type 1 reference coordinates. We also added per-model answer rates and clarified that MDErr is conditional on answered queries, whereas Success@τ uses all queries and counts unanswered cases as failures.

Response to Reviewer qXUw

  • Establishing the need for adaptive multi-scale reasoning.

We added both SearchOnly and FixedPipeline baselines. The adaptive framework consistently outperforms SearchOnly and achieves better fine-grained localization than the fixed pipeline at S@25 and S@50, while using approximately 9 rather than 25 tool calls per query. A sampled token-cost analysis further shows that FixedPipeline is substantially more expensive because it repeatedly accumulates and re-feeds images.

  • Strengthening Type 2 evaluation with human validation.

We now separately report Satisfies, Fails, and Indeterminate outcomes. Even under the stricter definition, the direct-contradiction rate remains 31.4–36.6%, so the conclusion about insufficient visual grounding does not depend on Indeterminate cases. We additionally conducted query-level human evaluation, covering all eight cities and seven languages. Human and automated-judge rankings are highly consistent, with Spearman’s ρ = 0.93.

Request for independent reassessment

Given the substantial new evidence, the corrections already incorporated into the manuscript, and the reviewers’ shared recognition of the benchmark’s practical value, we respectfully ask that the paper be assessed on the basis of its revised content and the full rebuttal evidence, rather than relying primarily on the original scores.

We sincerely thank the Program Committee and reviewers for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.