- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes GeoEvolve, a framework for image geolocalization that iteratively optimizes an external natural-language skill, GeoSkill, to improve hypothesis formation, tool use, evidence verification, and stopping decisions. The authors first use a frozen Qwen3.5-9B model equipped with web search and image retrieval to generate geolocalization trajectories. GLM-5.3-Flash then analyzes effective and harmful decisions within these trajectories to construct and progressively revise the skill, retaining useful updates through evaluation on a development set. This stage does not modify the execution model’s parameters. The authors further fine-tune Qwen3.5-9B on 5,000 skill-guided trajectories to obtain GeoEvolve-9B. Experiments on GEO-EVO-TEST and IM2GPS3K evaluate the effects of skill guidance, recursive evolution, and trajectory-based fine-tuning. On GEO-EVO-TEST, accuracy within 1 km increases from 14.61% with tools alone to 20.40% with GeoSkill, and further to 22.87%*after fine-tuning. The central contribution is to transform experience from individual geolocalization trajectories into an explicit policy that can be continually refined, reused, and used to support model training.
Critical strengths
- Tool-use experience is stored in an inspectable textual policy, allowing updates without repeatedly fine-tuning the execution model.
- The framework attempts to distinguish beneficial actions, non-compliance with existing rules, deficiencies in the rules themselves, and external tool failures, providing more fine-grained feedback than judging entire trajectories solely by their final outcomes.
Critical weaknesses
-
Insufficient discussion of and comparison with closely related work.
The paper does not cite Skill-Conditioned Visual Geolocation for Vision-Language Models, which already explores external natural-language skills, autonomous skill evolution driven by successful and failed trajectories, and policy improvement without updating model parameters for visual geolocation. More broadly, the coverage and citation of relevant geolocation research are insufficient. -
Contradictory descriptions of the experimental protocol.
The main text repeatedly emphasizes that GeoSkill is evolved once using trajectories from Qwen3.5-9B and then applied to other models without modification. However, the footnote of Table 3 on page 8 explicitly states: “Each GeoSkill is evolved separately for its corresponding base model.” Directly transferring a shared skill and evolving a separate skill for each model are fundamentally different experimental settings. This inconsistency directly affects the interpretation and validity of the paper’s conclusions. -
Insufficient experimental validation of a central claim.
The authors emphasize process-level attribution, but the existing ablations mainly examine tool availability, cold-start initialization, and subsequent evolution. They do not adequately compare the following alternatives under matched experimental conditions:- Updating skills based solely on final outcomes.
- Updating skills through standard trajectory reflection.
- Updating skills using the proposed process-level feedback.
Questions for authors
-
Which skill evolution protocol was actually used in the cross-model experiments?
The main text states that the same GeoSkill is transferred to other models without modification, whereas the footnote of Table 3 states that a separate skill is evolved for each model. Which setting was actually used? Could the authors report results for both direct transfer and separate evolution for each target model? -
Does process-level feedback outperform standard trajectory reflection?
Under the same initial skill, evolution data, and computational budget, could the authors compare updates based solely on final outcomes, standard trajectory reflection, and the proposed process-level feedback? How do the authors validate that the feedback correctly distinguishes failures to follow the skill instructions, flaws in the skill rules, and external tool failures? -
What denominator and aggregation procedure are used to compute accuracy?
GEO-EVO-TEST contains 1,000 samples. If each sample is evaluated once, accuracy should change in increments of 0.1 percentage points. How were values such as 6.56% and 14.61% calculated? In addition, Figure 3 identifies the results with zero on-policy trajectories as those of the cold-start skill, yet some threshold accuracies appear substantially different from the cold-start row in Table 2. For example, Table 2 reports an accuracy of 33.91% at 25 km, whereas the corresponding starting point in Figure 3 appears to be approximately 39%–40%. Could the authors clarify this discrepancy? -
Could the authors provide the complete information needed to reproduce skill evolution?
Could the authors release the final GeoSkill, the changes between key versions, the evolution prompts, the development set size, the update batch size, and the acceptance and rollback criteria? In particular, how is an update accepted or rejected when performance improves at some distance thresholds but declines at others?
I would be willing to raise my score if the authors satisfactorily address these concerns through clear explanations and supporting evidence.
2 comments
Clarification of Evaluation Metrics and Experimental Results (Q3)
Accuracy Calculation. Although Geo-EVO-Test contains 1,000 images, localization accuracy is calculated only over successfully completed runs. A run is considered completed if a valid geographic coordinate (latitude and longitude) can be extracted from its final output without encountering model request errors or tool execution failures. Specifically, runs are classified as incomplete in the following cases:
- Model request errors: Timeouts or API errors caused by requesting the underlying model.
- Tool execution failures: Errors or exceptions occurring when calling tools.
- Failure to extract valid coordinates: The final output lacks parseable latitude and longitude format.
Accuracy is computed over completed runs using the Haversine distance between predicted and reference coordinates at thresholds of 1, 25, 200, 750, and 2,500 km. Since incomplete runs are excluded, the evaluation denominator may be smaller than the 1,000 images in Geo-EVO-Test, explaining values such as 6.56% and 14.61%. We have clarified the completion criteria and accuracy calculation procedure in the revised manuscript.
Figure 3 vs. Table 2. As stated in Section 4.1, “a separate development split is used for checkpoint selection”. Figure 3 reports the performance of successive GeoSkill checkpoints on this development split, whereas Table 2 reports results on Geo-EVO-Test. Thus, the numerical differences arise from different evaluation splits rather than inconsistent experimental results. We have updated the caption of Figure 3 in the revised manuscript to explicitly clarify this distinction.
Reproducibility and Skill Update Criteria (Q4)
We thank the reviewer for emphasizing reproducibility. The complete implementation details, including the skill evolution and update procedures, are available in our anonymous repository (https://anonymous.4open.science/r/geoevolve-5E5F/).
Skill Revision and Validation. During on-policy evolution, each trajectory batch produces a process-level feedback report, which guides a conservative revision of the current GeoSkill while preserving unaffected policy rules. Each candidate is then evaluated on a fixed, held-out development split against both the initial cold-start skill and the currently accepted checkpoint.
Acceptance and Rollback Criteria. For checkpoint selection, accuracy is calculated over all development images, counting missing or invalid predictions as failures to ensure consistent comparisons. We evaluate each candidate using a weighted combination of localization accuracies at 1, 25, 200, 750, and 2500 km. A candidate is accepted only if it improves the overall performance over the currently accepted checkpoint while satisfying regression safeguards relative to the initial skill. These safeguards prevent substantial degradation at individual geographic scales, protect previously successful cases, and limit severe regressions and invalid predictions. Otherwise, the candidate is rejected, and the previous checkpoint is retained.
We thank the reviewer again for the valuable feedback. We have incorporated the additional experimental results and corresponding discussions into the revised manuscript. The code, data, and evolved skills are also publicly available at https://anonymous.4open.science/r/geoevolve-5E5F/ to facilitate reproducibility. We welcome any remaining concerns and would be happy to discuss them further. If our responses have satisfactorily addressed your concerns, we would greatly appreciate your reconsideration of the rating.
We thank the reviewer for the careful evaluation and constructive feedback, particularly the recognition of our inspectable textual skill and process-level feedback. We address the concerns below.
We thank the reviewer for highlighting [1]. Both works evolve natural-language skills for visual geolocalization, but differ in their skill representations and evolution mechanisms. The referenced work maintains a Skill-Graph of atomic skills that are retrieved and composed for geographic reasoning, whereas GeoEvolve recursively refines a unified GeoSkill for orchestrating hypothesis formation, tool selection, evidence verification, and stopping.
While [1] synthesizes, merges, and prunes atomic skills using successful and failed trajectories, GeoEvolve uses process-level credit assignment to distinguish agent non-compliance, policy deficiencies, and external tool failures, enabling targeted policy revisions. We have added [1] and clarified these distinctions in Related Work.
[1] Skill-Conditioned Visual Geolocation for Vision-Language Models, arXiv 2026
We apologize for the inconsistent Table 3 footnote, which has been corrected. GeoSkill was evolved solely from Qwen3.5-9B trajectories and transferred unchanged to Qwen3.5-27B and GLM-5.3-Flash, without model-specific evolution. The correction does not affect the results. Our experiments specifically assess cross-model reuse of a single skill. Separately evolving a skill for each backbone is a different setting beyond the scope of this evaluation.
We compare GeoEvolve with Outcome-only Evolution and Reflexion [1], representing outcome-based skill refinement and trajectory reflection, respectively, alongside ExpeL [2], TextGrad [3], and GEPA [4]. All methods use Qwen3.5-9B, identical tools, and comparable computational budgets on Geo-EVO-Test.
Effectiveness of Process-Level Feedback. GeoEvolve consistently outperforms Outcome-only Evolution, albeit by modest margins, and achieves the highest accuracy at four of five thresholds among the compared methods, with larger advantages at coarser scales. These results support the value of process-level feedback beyond outcome-based refinement and general-purpose self-improvement.
[1] Reflexion: Language Agents with Verbal Reinforcement Learning, NeurIPS 2023
[2] ExpeL: LLM Agents Are Experiential Learners, AAAI 2024
[3] Optimizing Generative AI by Backpropagating Language Model Feedback, Nature 2025
[4] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning, ICLR 2026
Qualitative Analysis of Failure Attribution and Skill Revision. We further trace actual GeoSkill revisions across accepted checkpoints to illustrate how process-level feedback leads to targeted policy updates.
(1) Geographic Evidence Verification.
(2) External Tool Failures.
(3) Refining the Stopping Policy.
These examples illustrate how GeoEvolve distinguishes agent non-compliance, policy deficiencies, and external tool failures to guide targeted revisions of geographic investigation rules rather than indiscriminately updating GeoSkill. Together with the quantitative comparisons, they provide further evidence for the benefits of process-level credit assignment.