- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces GeoGuesser Bench, a new benchmark for interactive visual geolocation. It comprises 282 tasks in an offline Street View environment, where models can rotate, zoom, and move to perform three types of tasks: precise localization, place naming, and navigation-based object finding and counting. The authors evaluate 23 vision–language models (VLMs), using an exponential distance-based score for precise localization and accuracy for the other two tasks. The results show that models with strong localization performance are not necessarily effective at active exploration. The paper also introduces a Planner–Worker–Compressor framework that substantially reduces the planner’s token consumption. Its main contribution lies in revealing that performance advantages on one type of geolocation task do not necessarily carry over to others. However, a substantial portion of its task design overlaps with prior work that has explored similar settings and introduced related benchmarks. The paper’s analysis of failure causes also warrants further examination.
Critical strengths
-
By evaluating localization and active information seeking within a unified framework, the benchmark reveals weaknesses that localization-only leaderboards may overlook.
-
The ablation studies show that allowing more actions does not always improve performance; many models even achieve higher localization scores when movement is disabled. These findings offer useful insights for improving future geolocation agents.
Critical weaknesses
1. The related-work discussion is seriously inadequate.
A key contribution of a new benchmark should be to identify capabilities that existing benchmarks cannot adequately evaluate and explain how the proposed benchmark addresses these gaps. However, existing work, including Embodied Web Agents [1], ERGeoBench [2], and the GeoAgent benchmark [3], already collectively covers substantial aspects of street-view geolocation through movement, active viewpoint selection, and capability diagnosis. These closely related papers are not cited in the manuscript, leaving the benchmark's distinct contribution insufficiently established.
Furthermore, the discussion of agentic geolocation and geolocation reasoning methods is incomplete. Relevant work such as Thinking with Map [4], GeoVista [5], and GeoAgent (CVPR 2026) [6] is neither cited nor analyzed. The authors should discuss these works and clarify how the proposed benchmark evaluates or extends the capabilities studied in prior research.
2. The evaluation results raise concerns about potential confounding between model capabilities and execution failures.
The paper explicitly reports that many zero scores result from early abstention, failure to provide an answer within the allotted budget, API errors, or missing valid final responses. Assigning zero scores to these outcomes is reasonable for an end-to-end agent evaluation. However, the frequency of these failures raises concerns about whether the reported performance primarily reflects model limitations or is substantially affected by issues in the evaluation framework and its handling of model APIs.
Abstention and budget management are legitimate aspects of agent capability, but they should be clearly distinguished from technical execution failures. Although the paper provides a breakdown of failure outcomes, further validation is needed to establish how much these technical issues affect the results. Such confounding could materially compromise the fairness and interpretation of comparisons across models.
3. The manuscript provides conflicting accounts of the failure-annotation procedure.
Three descriptions in the current manuscript are difficult to reconcile:
- Page 7 states that 160 failed episodes were manually labeled, while the remaining 705 were labeled by LLM-based classifiers.
- Page 16 states that five annotators labeled all 865 failed episodes and reports 95% agreement on the 300 episodes labeled by more than one annotator.
- The AI Use Statement on page 10 explicitly states that generative AI was not used to label failure modes.
The authors must reconcile these descriptions and clarify how the final labels were produced, whether automatically generated labels were reviewed by humans, and how annotation agreement was calculated. This inconsistency directly undermines the credibility and reproducibility of the failure analysis.
4. The benchmark's limited scale and uneven coverage weaken its representativeness and the reliability of fine-grained comparisons.
The benchmark contains only 282 evaluation instances across multiple tasks and splits. The geographic visualization suggests uneven coverage and concentrations in a limited set of regions, while the paper also acknowledges an urban and suburban bias in Place Naming and Navigation Challenge. Without a more detailed breakdown of geographic coverage, scene types, and task characteristics, the representativeness of the benchmark remains insufficiently established.
The small evaluation subsets also make performance comparisons sensitive to a handful of outcomes. On the 80-question Navigation Challenge, the highest accuracy of 35% corresponds to only two more correct answers than 32.5%. The street-level Place Naming subset contains just 11 questions, so a single answer changes accuracy by approximately 9.1 percentage points. These sample sizes limit the strength of conclusions about small performance differences and fine-grained capability rankings, motivating a larger and more systematically stratified evaluation.
References
[1] Yining Hong et al. Embodied Web Agents: Bridging Physical-Digital Realms for Integrated Agent Intelligence. 2025.
[2] Kaiwen Xue et al. ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models.
[3] Arka Mukherjee et al. GeoAgent: Evaluating VLM Geolocalization Through Embodied Navigation. 2026.
[4] Yuxiang Ji et al. Thinking with Map: Reinforced Parallel Map-Augmented Agent for Geolocalization. 2026.
[5] Yikun Wang et al. GeoVista: Web-Augmented Agentic Visual Reasoning for Geolocalization. 2025.
[6] Modi Jin et al. GeoAgent: Learning to Geolocate Everywhere with Reinforced Geographic Characteristics. 2026.
Questions for authors
-
Why report score-equivalent distance without also reporting actual geographic error?
The score-equivalent distance is a one-to-one transformation of the mean exponential score and therefore conveys no additional information beyond the score itself. Why not also report actual geographic error statistics?
-
Who generated the failure labels, and how was the reported 95% agreement calculated?
Page 7 states that 160 episodes were manually labeled and the remaining 705 were classified by LLMs. Appendix E states that five annotators labeled all 865 episodes. Meanwhile, the AI Use Statement explicitly denies using generative AI to label failure modes. Please clarify the annotation procedure ultimately used, including whether the annotators were human, whether automatically generated labels were reviewed by humans, and how the 95% agreement was calculated. Which version of the labels was used to produce the reported failure-mode statistics?
-
How substantially did technical failures affect the model rankings?
The paper reports that 93 episodes in the main evaluation terminated because requests exceeded provider size limits. How are these failures distributed across individual models and tasks? Can the authors address these potentially avoidable failures and assess their impact on the reported rankings?
-
Can the authors provide sufficient implementation and experimental details to reproduce and validate the Planner–Worker–Compressor framework?
The current description leaves important details unspecified. For example, how was the Worker fine-tuned, what training data were used, and what were the training settings?
-
How was the 24.39% Navigation Challenge accuracy in Table 4 calculated?
If the evaluation consists of a single run over all 80 questions, accuracy should change in increments of 1.25 percentage points. The reported 24.39% does not correspond to an integer number of correct answers under this setting. Please clarify how this result was obtained.
2 comments
Thank you for your responses and clarifications. While I appreciate your efforts, my concerns have not been fully addressed, so I have decided to keep my score unchanged.
Thank you for the careful review. Below we summarize our clarifications, updated results, and revisions.
Q1. Positioning relative to prior benchmarks and agentic methods
Static and Search-Augmented Geolocation. Static methods like GeoAgent (Jin et al., 2026; GeoSeek) predict locations from one fixed image, testing memorization rather than active exploration when initial views are uninformative. Agentic methods like GeoVista (zoom + web search), Thinking with Map (map/POI queries), and Embodied Web Agents (Street View + web search) rely on external retrieval, bypassing purely visual spatial exploration.
Interactive Street View Geolocation. ERGeoBench restricts actions to yaw/pitch/zoom within a single panorama, preventing street traversal when cues are occluded. Embodied navigation benchmarks either target destination/POI arrival (e.g., CityNav, WalkerBench, V-IRL) or, in GeoAgent (Mukherjee et al., 2026) and WanderBench, permit text descriptions and external Google Geocoding rather than direct coordinates in every episode while omitting local scene-fact queries. GeoGuesser Bench resolves all of these gaps in one offline, multi-panorama environment (external search/maps disabled) by jointly evaluating direct coordinate localization, Place Naming, and active Finding/Counting—showing localization consistency (0.97) fails to transfer to exploratory information seeking (0.39).
Q2. Interaction protocol and API request recovery
Technical API errors are separated from agent behaviors like abstention or budget exhaustion. We retried failed calls—recovering all 93 request-size-limit episodes—and updated the main table. Models share one interface, a 40-response cap, and default thinking effort, with token usage reported separately rather than capped per episode so reasoning models are not truncated.
Q3. Failure-mode annotation workflow
The discrepancy across Section 5.2, Appendix E, and the AI Use Statement came from an incomplete update across two stages. We tested LLM labeling (the outdated page 7 reference to 160 manual and 705 LLM labels), found its quality insufficient, and replaced it with manual annotation: five human raters labeled all 865 failed episodes, reaching 95% agreement across 300 doubly annotated episodes. Final statistics use only these human labels, and we align all three sections.
Q4. Benchmark scale, coverage, and statistical intervals
The 282 verified tasks span 96 countries/territories (Natural Earth), balancing ground-truth rigor with multimodal cost: Synthetic Localization (n=80) covers six continents evenly (11–15 each), Place Naming (n=80) spans four granularities (11 street, 31 city/town, 11 state/province, 27 country), and Navigation Challenge (n=80) has 52 Finding and 28 Counting tasks in urban/suburban streetscapes. Alongside Table 1's standard errors across 23 models, we add regional tables and 95% CIs, focusing on broad cross-task shifts over small gaps.
Q5. Localization metrics and geodesic distance statistics
Alongside the GeoGuessr score, we add WGS84 mean/median errors (on valid coordinates), valid counts, and all-task Acc@25 km (abstentions/invalid = 0). Across evaluated models, frontier models achieve city-scale median precision on answered episodes while rare cross-region misses inflate mean errors by an order of magnitude; under static/no-movement settings, models that commit from initial cues retain strong localization whereas cautious models often abstain. For example, on our 80-task static single-view baseline, Claude Opus 4.6 returns 80/80 valid coordinates (3,413.3/5000, 28.8% Acc@25 km) with 224.8 km median versus 1,629.0 km mean error, while GPT-5.2 abstains on 78/80 items (2/80 valid, 68.7 score, 1,722.1 km, 1.25% Acc@25 km); in interactive Synthetic runs (n=80), GPT-5.2 achieves 1.4 km median versus 368.7 km mean error (23/80 valid, 13/80 Acc@25 km) and Qwen3.5-397B achieves 222.7 versus 1,707.4 km (55/80 valid, 20/80 Acc@25 km).
Q6. Planner–Worker–Compressor implementation details
In our framework ablation (separate from the 40-response main protocol), episodes run up to 20 Planner rounds with up to five Worker steps per round and a mandatory final answer. Given the view and Planner instruction, the Qwen2.5-VL-7B Worker predicts one of ten actions (forward, small/large left/right turn, look up/down, zoom in/out, stop). Between rounds, the Compressor summarizes the question, recent actions, and key-frame notes into a
briefandfocus_hintfor the Planner.Q7. Table 4 Navigation Challenge accuracy calculation
The 24.39% in Table 4 came from two blank summary rows appended to the 80-task CSV (20/82 = 24.39%). Removing them corrects both conditions to 20/80 = 25.00% (scaling by 82/80 = 1.025, leaving predictions and tie unchanged), while cutting recorded API usage across the 80 tasks from 10.91M to 4.69M tokens.