acceptodds
Under review as a conference paper at ICLR 2027

Geoguesser Bench: Benchmarking Vision–Language Agents on Interactive Geolocation

Abstract

Vision–language model (VLM) agents that act in the open world need two abilities: world knowledge to interpret what they see, and exploration to find what they cannot yet see. We study how agents combine the two through interactive visual geolocation. GeoGuesser Bench contains 282 tasks in an offline Street View environment where agents rotate, zoom, and move between real panoramas. Its three task types shift the balance from knowledge to exploration: agents (1) report the coordinates of their starting location, (2) name the enclosing street, city, state, or country, or (3) find, read, and count nearby objects. The coordinate tasks in (1) also include 42 real GeoGuessr World Championship rounds, whose expert guesses provide a human reference. We evaluate 23 VLMs: the best accuracy is 62.5% on place naming and 35.0% on object finding, and the best agent averages 4352.4 GeoGuessr points on the championship rounds, against 4770.3 for the better expert in each round. Beyond these headline numbers, world knowledge and exploration are only weakly correlated across models: agents that localize well do not necessarily explore well. To understand where agents fail, we categorize their errors into vision, navigation, and tool-use failures. Further ablations show that the strongest agents, such as Claude Opus 4.8, still perform well with limited views, while weaker agents degrade; customizing the environment thus adjusts task difficulty. The environment’s verifiable rewards and restorable state also support training, and our Planner–Worker–Compressor harness cuts planner tokens by 57–61% by compacting the interaction history. We hope GeoGuesser Bench and its environment serve as a high-quality foundation for simulating real-world exploration at scale.

Primary area:
datasets and benchmarks
Keywords:
interactive visual geolocation, vision-language agents, multimodal reasoning, active perception, visual navigation, tool use, agent evaluation, benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_142JOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GeoGuesser Bench, a new benchmark for interactive visual geolocation. It comprises 282 tasks in an offline Street View environment, where models can rotate, zoom, and move to perform three types of tasks: precise localization, place naming, and navigation-based object finding and counting. The authors evaluate 23 vision–language models (VLMs), using an exponential distance-based score for precise localization and accuracy for the other two tasks. The results show that models with strong localization performance are not necessarily effective at active exploration. The paper also introduces a Planner–Worker–Compressor framework that substantially reduces the planner’s token consumption. Its main contribution lies in revealing that performance advantages on one type of geolocation task do not necessarily carry over to others. However, a substantial portion of its task design overlaps with prior work that has explored similar settings and introduced related benchmarks. The paper’s analysis of failure causes also warrants further examination.

Critical strengths
  • By evaluating localization and active information seeking within a unified framework, the benchmark reveals weaknesses that localization-only leaderboards may overlook.

  • The ablation studies show that allowing more actions does not always improve performance; many models even achieve higher localization scores when movement is disabled. These findings offer useful insights for improving future geolocation agents.

Critical weaknesses

1. The related-work discussion is seriously inadequate.

A key contribution of a new benchmark should be to identify capabilities that existing benchmarks cannot adequately evaluate and explain how the proposed benchmark addresses these gaps. However, existing work, including Embodied Web Agents [1], ERGeoBench [2], and the GeoAgent benchmark [3], already collectively covers substantial aspects of street-view geolocation through movement, active viewpoint selection, and capability diagnosis. These closely related papers are not cited in the manuscript, leaving the benchmark's distinct contribution insufficiently established.

Furthermore, the discussion of agentic geolocation and geolocation reasoning methods is incomplete. Relevant work such as Thinking with Map [4], GeoVista [5], and GeoAgent (CVPR 2026) [6] is neither cited nor analyzed. The authors should discuss these works and clarify how the proposed benchmark evaluates or extends the capabilities studied in prior research.

2. The evaluation results raise concerns about potential confounding between model capabilities and execution failures.

The paper explicitly reports that many zero scores result from early abstention, failure to provide an answer within the allotted budget, API errors, or missing valid final responses. Assigning zero scores to these outcomes is reasonable for an end-to-end agent evaluation. However, the frequency of these failures raises concerns about whether the reported performance primarily reflects model limitations or is substantially affected by issues in the evaluation framework and its handling of model APIs.

Abstention and budget management are legitimate aspects of agent capability, but they should be clearly distinguished from technical execution failures. Although the paper provides a breakdown of failure outcomes, further validation is needed to establish how much these technical issues affect the results. Such confounding could materially compromise the fairness and interpretation of comparisons across models.

3. The manuscript provides conflicting accounts of the failure-annotation procedure.

Three descriptions in the current manuscript are difficult to reconcile:

  • Page 7 states that 160 failed episodes were manually labeled, while the remaining 705 were labeled by LLM-based classifiers.
  • Page 16 states that five annotators labeled all 865 failed episodes and reports 95% agreement on the 300 episodes labeled by more than one annotator.
  • The AI Use Statement on page 10 explicitly states that generative AI was not used to label failure modes.

The authors must reconcile these descriptions and clarify how the final labels were produced, whether automatically generated labels were reviewed by humans, and how annotation agreement was calculated. This inconsistency directly undermines the credibility and reproducibility of the failure analysis.

4. The benchmark's limited scale and uneven coverage weaken its representativeness and the reliability of fine-grained comparisons.

The benchmark contains only 282 evaluation instances across multiple tasks and splits. The geographic visualization suggests uneven coverage and concentrations in a limited set of regions, while the paper also acknowledges an urban and suburban bias in Place Naming and Navigation Challenge. Without a more detailed breakdown of geographic coverage, scene types, and task characteristics, the representativeness of the benchmark remains insufficiently established.

The small evaluation subsets also make performance comparisons sensitive to a handful of outcomes. On the 80-question Navigation Challenge, the highest accuracy of 35% corresponds to only two more correct answers than 32.5%. The street-level Place Naming subset contains just 11 questions, so a single answer changes accuracy by approximately 9.1 percentage points. These sample sizes limit the strength of conclusions about small performance differences and fine-grained capability rankings, motivating a larger and more systematically stratified evaluation.

References

[1] Yining Hong et al. Embodied Web Agents: Bridging Physical-Digital Realms for Integrated Agent Intelligence. 2025.

[2] Kaiwen Xue et al. ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models.

[3] Arka Mukherjee et al. GeoAgent: Evaluating VLM Geolocalization Through Embodied Navigation. 2026.

[4] Yuxiang Ji et al. Thinking with Map: Reinforced Parallel Map-Augmented Agent for Geolocalization. 2026.

[5] Yikun Wang et al. GeoVista: Web-Augmented Agentic Visual Reasoning for Geolocalization. 2025.

[6] Modi Jin et al. GeoAgent: Learning to Geolocate Everywhere with Reinforced Geographic Characteristics. 2026.

Questions for authors
  1. Why report score-equivalent distance without also reporting actual geographic error?

    The score-equivalent distance is a one-to-one transformation of the mean exponential score and therefore conveys no additional information beyond the score itself. Why not also report actual geographic error statistics?

  2. Who generated the failure labels, and how was the reported 95% agreement calculated?

    Page 7 states that 160 episodes were manually labeled and the remaining 705 were classified by LLMs. Appendix E states that five annotators labeled all 865 episodes. Meanwhile, the AI Use Statement explicitly denies using generative AI to label failure modes. Please clarify the annotation procedure ultimately used, including whether the annotators were human, whether automatically generated labels were reviewed by humans, and how the 95% agreement was calculated. Which version of the labels was used to produce the reported failure-mode statistics?

  3. How substantially did technical failures affect the model rankings?

    The paper reports that 93 episodes in the main evaluation terminated because requests exceeded provider size limits. How are these failures distributed across individual models and tasks? Can the authors address these potentially avoidable failures and assess their impact on the reported rankings?

  4. Can the authors provide sufficient implementation and experimental details to reproduce and validate the Planner–Worker–Compressor framework?

    The current description leaves important details unspecified. For example, how was the Worker fine-tuned, what training data were used, and what were the training settings?

  5. How was the 24.39% Navigation Challenge accuracy in Table 4 calculated?

    If the evaluation consists of a single run over all 80 questions, accuracy should change in increments of 1.25 percentage points. The reported 24.39% does not correspond to an integer number of correct answers under this setting. Please clarify how this result was obtained.

2 comments

Thank you for the careful review. Below we summarize our clarifications, updated results, and revisions.

Q1. Positioning relative to prior benchmarks and agentic methods

Static and Search-Augmented Geolocation. Static methods like GeoAgent (Jin et al., 2026; GeoSeek) predict locations from one fixed image, testing memorization rather than active exploration when initial views are uninformative. Agentic methods like GeoVista (zoom + web search), Thinking with Map (map/POI queries), and Embodied Web Agents (Street View + web search) rely on external retrieval, bypassing purely visual spatial exploration.

Interactive Street View Geolocation. ERGeoBench restricts actions to yaw/pitch/zoom within a single panorama, preventing street traversal when cues are occluded. Embodied navigation benchmarks either target destination/POI arrival (e.g., CityNav, WalkerBench, V-IRL) or, in GeoAgent (Mukherjee et al., 2026) and WanderBench, permit text descriptions and external Google Geocoding rather than direct coordinates in every episode while omitting local scene-fact queries. GeoGuesser Bench resolves all of these gaps in one offline, multi-panorama environment (external search/maps disabled) by jointly evaluating direct coordinate localization, Place Naming, and active Finding/Counting—showing localization consistency (0.97) fails to transfer to exploratory information seeking (0.39).

Q2. Interaction protocol and API request recovery

Technical API errors are separated from agent behaviors like abstention or budget exhaustion. We retried failed calls—recovering all 93 request-size-limit episodes—and updated the main table. Models share one interface, a 40-response cap, and default thinking effort, with token usage reported separately rather than capped per episode so reasoning models are not truncated.

Q3. Failure-mode annotation workflow

The discrepancy across Section 5.2, Appendix E, and the AI Use Statement came from an incomplete update across two stages. We tested LLM labeling (the outdated page 7 reference to 160 manual and 705 LLM labels), found its quality insufficient, and replaced it with manual annotation: five human raters labeled all 865 failed episodes, reaching 95% agreement across 300 doubly annotated episodes. Final statistics use only these human labels, and we align all three sections.

Q4. Benchmark scale, coverage, and statistical intervals

The 282 verified tasks span 96 countries/territories (Natural Earth), balancing ground-truth rigor with multimodal cost: Synthetic Localization (n=80) covers six continents evenly (11–15 each), Place Naming (n=80) spans four granularities (11 street, 31 city/town, 11 state/province, 27 country), and Navigation Challenge (n=80) has 52 Finding and 28 Counting tasks in urban/suburban streetscapes. Alongside Table 1's standard errors across 23 models, we add regional tables and 95% CIs, focusing on broad cross-task shifts over small gaps.

Q5. Localization metrics and geodesic distance statistics

Alongside the GeoGuessr score, we add WGS84 mean/median errors (on valid coordinates), valid counts, and all-task Acc@25 km (abstentions/invalid = 0). Across evaluated models, frontier models achieve city-scale median precision on answered episodes while rare cross-region misses inflate mean errors by an order of magnitude; under static/no-movement settings, models that commit from initial cues retain strong localization whereas cautious models often abstain. For example, on our 80-task static single-view baseline, Claude Opus 4.6 returns 80/80 valid coordinates (3,413.3/5000, 28.8% Acc@25 km) with 224.8 km median versus 1,629.0 km mean error, while GPT-5.2 abstains on 78/80 items (2/80 valid, 68.7 score, 1,722.1 km, 1.25% Acc@25 km); in interactive Synthetic runs (n=80), GPT-5.2 achieves 1.4 km median versus 368.7 km mean error (23/80 valid, 13/80 Acc@25 km) and Qwen3.5-397B achieves 222.7 versus 1,707.4 km (55/80 valid, 20/80 Acc@25 km).

Q6. Planner–Worker–Compressor implementation details

In our framework ablation (separate from the 40-response main protocol), episodes run up to 20 Planner rounds with up to five Worker steps per round and a mandatory final answer. Given the view and Planner instruction, the Qwen2.5-VL-7B Worker predicts one of ten actions (forward, small/large left/right turn, look up/down, zoom in/out, stop). Between rounds, the Compressor summarizes the question, recent actions, and key-frame notes into a brief and focus_hint for the Planner.

Q7. Table 4 Navigation Challenge accuracy calculation

The 24.39% in Table 4 came from two blank summary rows appended to the 80-task CSV (20/82 = 24.39%). Removing them corrects both conditions to 20/80 = 25.00% (scaling by 82/80 = 1.025, leaving predictions and tie unchanged), while cutting recorded API usage across the 80 tasks from 10.91M to 4.69M tokens.

Thank you for your responses and clarifications. While I appreciate your efforts, my concerns have not been fully addressed, so I have decided to keep my score unchanged.

Reviewer_fZcZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces GeoGuesser Bench an interactive benchmark with 282 tasks built using Google Street View panoramas. The tasks cover three settings: predicting geographic coordinates, identifying locations at different levels of granularity, and navigating to find or inspect objects in the environment. The authors evaluate multiple VLMs and find that models performing well on geolocation do not necessarily perform well on tasks requiring exploration. They also compare model performance against expert GeoGuessr players on 42 competition rounds, and introduce a Planner–Worker–Compressor framework to reduce token usage.

Critical strengths

If the dataset can be legally released as a fully offline environment, it could provide a useful benchmark for evaluating visual grounding and exploration in real-world settings.

Critical weaknesses
  1. Major Concern: Section 4 states that Google Street View panoramas and navigation links are bundled into an offline environment, which the authors intend to release. But Google's standard terms restrict downloading, caching, and redistributing Street View imagery. It is unclear how the authors plan to publicly release this dataset. Have they obtained explicit permission from Google to distribute the panoramas and use them for model evaluation? If the release includes only panorama IDs and annotations, how will the authors ensure that the benchmark remains reproducible and fully offline, as claimed?

  2. Lack of relevant related work. The paper misses three closely related works: [1] Can MLLMs Find Their Way in a City? Exploring Emergent Navigation from Web-Scale Knowledge (EACL 2026) [2] V-IRL: Grounding Virtual Intelligence in Real Life (ECCV 2024) [3] ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models

The papers above study vision-language agents operating in real-world urban environments (specifically Google Street View) with a focus on visual grounding and navigation.

Please refer to the compliance policy mentioned in the papers above to understand Google Street View rules.

  1. The paper identifies navigational reasoning as the most common failure mode, but groups poor navigation decisions together with cases where agents lose track of previously observed objects or explored areas. For example, Figure 11 shows an agent failing to count objects it encountered earlier. It is unclear whether such failures result from poor navigation planning or memory. An experiment that provides a reliable record of visited locations and observed objects would help clarify the role of memory in exploration performance.
Questions for authors

Additional dataset release concern: Releasing only panorama IDs and annotations will pose reproducibility issues as Google explicitly states that panorama IDs can change or become invalid over time. If authors can address this problem, then the dataset is useful.

1 comment

Thank you for the constructive review. Below we summarize our clarifications and revisions on related work, failure-mode taxonomy, and dataset release.

Q1. Positioning relative to prior benchmarks and methods

Static and Search-Augmented Geolocation. Existing geolocation work largely predicts locations from fixed views or relies on external digital retrieval. Static methods like GeoAgent (Jin et al., 2026; GeoSeek) infer locations from one supplied image, testing static memorization rather than active exploration when initial views are uninformative. Recent agentic methods query external tools: GeoVista pairs image zoom with web search on GeoBench, Thinking with Map queries map/POI services on MAPBench, and Embodied Web Agents combines Street View movement with web search. However, search engines and map APIs bypass purely visual spatial reasoning, leaving unmeasured whether an agent can gather missing evidence solely by exploring the environment.

Urban Navigation and Grounding Benchmarks. Other works situate agents in Street View, but target route execution or local perception rather than global localization. CityNav evaluates route planning to target destinations, WalkerBench measures instruction-guided POI arrival (SR/SPL) and geometric estimation (height, distance, bearing), and V-IRL grounds agents across VLN, VQA, and place matching against nearby place records. Reaching a waypoint or matching local place records does not test inferring global starting coordinates from visual cues or exploring to answer unseen factual and counting queries.

Interactive Street View Geolocation. Closer benchmarks add viewpoint control for geolocation, yet remain limited in exploration scope, output precision, or task breadth. ERGeoBench restricts actions to yaw, pitch, and zoom within a single panorama, preventing movement down a street when cues are occluded. WanderBench and GeoAgent (Mukherjee et al., 2026) support multi-panorama traversal, but permit text descriptions or external Google Geocoding rather than direct coordinates in every episode, and evaluate localization without local scene-fact queries. GeoGuesser Bench resolves all of these gaps in one offline, multi-panorama environment (external search/maps disabled in the main setting) by jointly evaluating (i) direct coordinate prediction under continuous scoring, (ii) hierarchical Place Naming, and (iii) active Finding and Counting—showing that localization consistency (Spearman 0.97 across splits) fails to transfer to exploratory information seeking (0.39 with Navigation Challenge).

Q2. Role of memory and planning in exploratory tasks

FM2 spans both route-selection errors and failures to retain previously observed objects or visited views (as in Figure 11's counting trajectory), and we revise Section 5.2 to frame FM2 jointly as navigation and working memory rather than route planning alone. Because our main benchmark evaluates base VLMs end-to-end through a shared interface without separate memory and planning modules, utilizing prior observations is coupled with choosing subsequent actions; supplying an external log of visited views and observed objects therefore evaluates a memory-augmented harness rather than isolating an internal component of the base VLM, a distinction we make explicit in Section 5.2.

Q3. Google imagery and dataset release

We are currently in active discussions with the relevant parties regarding authorization for the use and distribution of a subset of the Street View imagery. We are confident that we will obtain authorization soon and be able to make this benchmark publicly accessible when this paper is accepted.

Reviewer_JrJhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an interactive benchmark to evaluate VLMs' ability on geolocation. The benchmarks include three key tasks: precise localization, placing naming and navigation challenge. The paper also provides comprehensive benchmarks and baseline results of recent state-of-the-art models.

Critical strengths
  • The proposed benchmark is interesting and well-motivated.

  • The paper provides most recent baseline methods and show interesting observations.

Critical weaknesses
  • My main concern is the overlap with existing geolocation benchmarks. I find two recent papers that also provide interactive benchmarks for geolocation problems including WanderBench (CVPR 2026) and GeoAgent (arXiv:2602.12617). I am not sure if we need another new benchmark for this task considering there are still a lot of this category of benchmarks.

  • I think the proposed navigation challenge, which requires models to understand and navigate their surrounding environment, is basically different from the other two tasks. It seems out of place in a geolocation benchmark, and combining all three tasks into a single benchmark may not be appropriate. Besides, there is already a benchmark named WalkerBench (ECCV 2026) that provides interactive environment for the task.

  • How about token budget / cost of each model? I think settings like "thinking effort" and tool availability (like web search, image search, etc.) would largely affect the final results. How do you unify the settings cross different models. I would also be interesting to show the cost / token usage for different models.

Questions for authors

Please see my comments in Weaknesses.

1 comment

Thank you for the constructive review. Below we clarify our positioning relative to recent benchmarks, the role of unified cross-task evaluation, and our standardized model settings and token usage.

Q1. Positioning relative to prior benchmarks and methods

Static and Search-Augmented Geolocation. Existing geolocation work largely predicts locations from fixed views or relies on external digital retrieval. Static methods like GeoAgent (Jin et al., 2026; GeoSeek) infer locations from one supplied image, testing static memorization rather than active exploration when initial views are uninformative. Recent agentic methods query external tools: GeoVista pairs image zoom with web search on GeoBench, Thinking with Map queries map/POI services on MAPBench, and Embodied Web Agents combines Street View movement with web search. However, search engines and map APIs bypass purely visual spatial reasoning, leaving unmeasured whether an agent can gather missing evidence solely by exploring the environment.

Urban Navigation and Grounding Benchmarks. Other works situate agents in Street View, but target route execution or local perception rather than global localization. CityNav evaluates route planning to target destinations, WalkerBench measures instruction-guided POI arrival (SR/SPL) and geometric estimation (height, distance, bearing), and V-IRL grounds agents across VLN, VQA, and place matching against nearby place records. Reaching a waypoint or matching local place records does not test inferring global starting coordinates from visual cues or exploring to answer unseen factual and counting queries.

Interactive Street View Geolocation. Closer benchmarks add viewpoint control for geolocation, yet remain limited in exploration scope, output precision, or task breadth. ERGeoBench restricts actions to yaw, pitch, and zoom within a single panorama, preventing movement down a street when cues are occluded. WanderBench and GeoAgent (Mukherjee et al., 2026) support multi-panorama traversal, but permit text descriptions or external Google Geocoding rather than direct coordinates in every episode, and evaluate localization without local scene-fact queries. GeoGuesser Bench resolves all of these gaps in one offline, multi-panorama environment (external search/maps disabled in the main setting) by jointly evaluating (i) direct coordinate prediction under continuous scoring, (ii) hierarchical Place Naming, and (iii) active Finding and Counting—showing that localization consistency (Spearman 0.97 across splits) fails to transfer to exploratory information seeking (0.39 with Navigation Challenge).

Q2. Role of Navigation Challenge in unified evaluation

Evaluating Precise Localization, Place Naming, and Navigation Challenge through one Street View interface tests whether recognizing where a scene is transfers to actively acquiring unseen local facts—such as business names, sign text, or object counts across panoramas—where even prompts with relative directions require identifying, reading, or counting visual evidence rather than merely reaching a waypoint. Retaining separate task-level scores exposes sharp rank reversals: in Table 1, GPT-5.4 scores 4,221.1 on Synthetic Localization versus 12.5% on Navigation Challenge, whereas Claude Fable 5 achieves 3,164.2 and 35.0%, showing why localization leaderboards alone capture an incomplete view of visual exploration.

Q3. Standardized protocol, tool settings, and token usage

All models in our main evaluation share identical prompts, camera/movement tools, and a 40 tool-calling-response cap (each response may invoke multiple tool calls), with all external tools—web search, image search, and map APIs—disabled except in the separate web-search ablation. Thinking effort stays at provider defaults while per-episode token usage is tracked separately: across evaluated models, token consumption scales primarily with multi-turn trajectory length (accumulating visual context) and reasoning verbosity rather than the step cap alone—for example, across all 282 tasks, mean input+output usage is 16.8k tokens/episode for Qwen3.5-122B, 32.0k for GPT-5.2, and 71.2k for Qwen3.5-397B. Likewise, across static and no-movement settings without multi-panorama traversal, models that commit from initial visual cues retain strong localization scores (Table 3) whereas cautious models drop sharply without exploration; for instance, on our 80-task static single-view baseline, Claude Opus 4.6 attains 3,413.3/5000 (80/80 valid coordinates) whereas GPT-5.2 abstains on 78/80 tasks (68.7/5000).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.