- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces IDOL, a benchmark for geographic localization from natural language instructions within a specified city. Agents access place search, satellite imagery, and street imagery to identify either a particular described location or locations satisfying visually verifiable conditions. The benchmark contains 7,126 query instances across eight cities and seven languages, derived from 1,018 original queries. Its main contributions are the curated dataset, a common framework for agent interaction with geospatial tools, and diagnostic evaluation protocols covering localization accuracy, candidate retrieval, and visual evidence supporting returned locations. Validation includes experiments with multiple multimodal language models, tool ablations for Type 1, comparisons across cities and languages, and human analysis of failure cases. For Type 2, the authors assess three automated judges against human annotations of 600 observed places. The results reveal substantial gaps between retrieving a correct candidate and selecting the final location, demonstrate benefits from visual verification, and highlight differences in how consistently models gather evidence to support their answers.
Critical strengths
Practically relevant task formulation and a useful benchmark resource. IDOL evaluates whether agents can resolve informal location requests within a known city, covering both specific targets and multiple valid locations satisfying visual conditions. Its 1,018 original queries span eight cities, undergo human review, and are provided in seven language versions. This combination of practical tasks, geographic coverage, and reviewed data provides a valuable testbed for studying how multimodal agents connect user instructions to real locations through geospatial tools. The benchmark’s potential usefulness to subsequent research is a substantive strength supporting acceptance. Then, the evaluation distinguishes candidate retrieval from final localization, making important performance gaps visible. For example, Gemini 3.5 Flash achieves 71.1% oracle success within 50 m but only 51.6% final success, revealing substantial room for improvement after candidate retrieval. Type 2 additionally tracks whether returned locations have supporting visual observations and satisfy the requested conditions. Tool ablations further show consistent accuracy losses when StreetView is removed across the six evaluated models. These analyses help identify concrete weaknesses in search, selection, and verification, substantially increasing the paper’s value as an evaluation contribution.
Critical weaknesses
The central motivation for multi-scale geospatial reasoning is insufficiently substantiated. The paper motivates IDOL through the need to fuse evidence across map, satellite, and street-level views, but does not establish how frequently its queries actually require such integration. The agent is given the city and can retrieve candidate names and coordinates, yet Table 7 omits a SearchPlaces-only baseline, leaving the additional difficulty introduced by visual constraints insufficiently characterized. The existing ablations support the usefulness of street-level verification, while the benefits of satellite imagery vary across models and metrics; they do not identify which queries require complementary evidence from different scales. Moreover, Type 2 success requires image observation by definition, so penalizing answers without visual observations cannot independently establish the necessity of visual verification. Without clearer controls distinguishing semantic retrieval, visual matching, and cross-scale reasoning, it remains uncertain whether IDOL specifically measures the capability that motivates its design. This weakens the justification for the benchmark’s central contribution.
Insufficient validation of the Type 2 evaluation and its interpretation. Requiring visual evidence is a reasonable benchmark objective, but the Type 2 success metric combines evidence acquisition with answer validity, leaving unsupported answers indistinguishable from incorrect ones in the headline score (Section 4.2). More critically, Section 5.6 interprets unsuccessful post-hoc visual verification as hallucination, although the judge explicitly permits an “indeterminate” outcome whose treatment is not clarified (Appendix A.8.4). The comparison of three judges and the 600-place human audit are valuable safeguards, but the primary judge achieves 78.7% human agreement and κ = 0.526 (Table 5), and the paper does not report human-validated query-level success rates or uncertainty estimates for the resulting comparisons. Consequently, the evidence does not sufficiently establish how much of the reported failure reflects incorrect locations, missing visual support, or judge error. Because reliable measurement is central to a benchmark contribution, this uncertainty materially weakens my confidence in the Type 2 comparisons and the conclusions about model hallucination.
Questions for authors
-
Clarification of unsuccessful verification and hallucination. How are the judge’s
failsandindeterminateverdicts treated in the subsequent verification analysis in Section 5.6 and Table 6? Could you report their separate frequencies and clarify whether the stated hallucination rate includes answers that cannot be confirmed because the imagery is insufficient, rather than because the location demonstrably violates the query? -
Additional evidence on the motivating capability. For Type 1, how well does an agent using only SearchPlaces perform? Across both task families, how does the adaptive agent compare with a fixed pipeline that first searches and then verifies, using the same backbone, available tools, and comparable call budgets? Do these comparisons support the claim that IDOL tests adaptive integration across scales beyond candidate retrieval and visual matching?
-
Additional validation of the Type 2 comparisons. On a representative subset of queries with complete answer sets, do human judgments of the same retrieved evidence reproduce the main differences in Type 2 Succ and the resulting model rankings? Could you provide uncertainty estimates for these comparisons, given the primary judge’s 78.7% agreement with human annotations in Table 5?
-
Clarification of consistency between the user query and the judging predicate. The agent receives a colloquial query, whereas the judge additionally receives a precise visual predicate, as described in Appendices A.8.1 and A.8.4. Did the human review explicitly verify that every condition in the predicate is expressed in, or unambiguously implied by, the query visible to the agent? How were cases admitting multiple reasonable interpretations handled?
-
Additional validation of sensitivity to alternative correct locations. Using the failure audit in Section 5.5, could you quantify how accepting the alternative matches confirmed by human reviewers would affect Type 1 success rates and the main model comparisons? What evidence indicates whether residual ground truth ambiguity materially affects the conclusions, particularly for models with similar scores?
-
Clarification of the conclusions about performance across languages. What checks established that translations into the other six languages preserve the visual and spatial clues, as well as the ambiguity, of the original Chinese queries? Given that the agent prompts require English search keywords, what evidence distinguishes the language effects reported in Section 5.4 from translation errors or differences in generating English retrieval queries?
Minor:The text in several figures, particularly Figures 1, is difficult to read at normal viewing size.
I am open to revising my score if the authors can provide satisfactory clarifications that address the concerns raised above.
2 comments
Clarification Regarding Potential Ethical Concerns
Dear Reviewer qXUw,
Thank you for your updated assessment. We are glad to hear that your main concerns have been addressed.
Regarding the potential ethical concerns mentioned in your comment, we currently cannot see any corresponding comment or specific concern in the discussion thread. The visible review also indicates “Flag For Ethics Review: No.” Therefore, we are unsure what issue may still require clarification or action from us.
If there is anything the authors should address, please let us know. We would be happy to provide any necessary clarification promptly.
Thank you again for your time and consideration.
Thanks for the detailed response. My main concerns have been addressed.
After the ac address the potential ethical concerns, I will change the score accrodingly.