acceptodds
Withdrawn from ICLR 2027

GeoSkill: Benchmarking and Evolving Domain Skills for Remote-Sensing Multimodal Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Ultra-high-resolution (UHR) remote-sensing interpretation requires coordinating local visual inspection and broader spatial analysis under task-specific constraints. Tool-augmented agents have advanced these capabilities, but effective execution does not itself make the underlying procedures explicitly reusable. Domain skills can preserve this expertise, yet their benefits and limitations in UHR tasks remain unclear. We introduce GeoSkillBench, comprising 1,385 evaluation questions across 13 UHR tasks, fine-grained skill annotations, a structured skill library, and a shared geospatial tool environment. With models and base tools held fixed, we compare agents without skill documents, with expert-curated skills, and with self-generated plans. Expert skills improve accuracy across the evaluated models, while their advantage over self-generated plans varies by model. Paired trajectories reveal gaps in spatial verification and tool execution. Building on these findings, we introduce GeoSkillAgent, which consolidates procedural corrections from failed trajectories on a dedicated training set to iteratively refine an expert-curated SkillBank. Candidate updates and the merged SkillBank are screened for validation accuracy gains and overall stability before adoption. Model parameters remain fixed throughout, and the final SkillBank is frozen for testing. Our refinement experiments show that learning from failed trajectories yields larger gains than learning from successful trajectories alone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Ultra-high-resolution remote sensing, Multimodal agents, Skill

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YJPYOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates reusable domain skills for tool-augmented multimodal agents in ultra-high-resolution remote-sensing interpretation. The authors introduce GeoSkillBench, comprising 1,385 questions across 13 tasks, skill annotations, a structured skill library, and shared geospatial tools. With models, tools, and interaction budgets held fixed, it compares agents without skill documents, with expert-curated skills, and with self-generated plans through paired outcomes and execution traces. The authors also propose GeoSkillAgent, which extracts procedural corrections from failed training trajectories to refine the skill library while keeping model parameters fixed. The experimental analysis indicates that the benefits of skill guidance depend on both the task and the executing model. Expert-curated skills do not consistently outperform self-generated plans, and aggregate gains are accompanied by regressions on some questions. The authors further report that refining the skill library using failure feedback improves overall accuracy under the evaluated settings.

Critical strengths

The authors provide a structured library of domain-specific skill documents and an accompanying evaluation dataset, with clearly defined task and skill categories and quality control combining automated screening with expert review. Beyond providing performance benchmarks, more analyses offer empirical insights into when domain skills help and when they may lead to regressions. Furthermore, the authors introduce a skill-library refinement process driven by failed execution trajectories and incorporating validation and stability checks. Under the reported experimental settings, this process achieves higher overall accuracy than the static expert-curated library, offering a useful direction for future research on skill-guided remote-sensing agents.

Critical weaknesses
  1. Insufficient detail on dataset construction and skill evolution limits quality assessment and reproducibility. Although the paper describes its data sources, automated screening, and 13 task categories, the rationale for this taxonomy, boundaries between overlapping categories, and assignment of questions requiring multiple capabilities remain insufficiently explained. The reported review by 10 experts also lacks details on their expertise, responsibilities, reviewers per question, and agreement on answers and skill annotations. These omissions make the reliability of human quality control difficult to assess. For GeoSkillAgent, despite the reported partition sizes and scene-level separation, the training pool’s composition by source, task, and construction route, together with its complete selection procedure, remains unclear. This limits independent verification of data quality and reproduction of the refinement process and results.
  2. The benchmark primarily measures four-option multiple-choice accuracy, supplemented by skill access, tool errors, interaction counts, and budget exhaustion. These statistics capture some execution problems but do not directly assess whether critical steps are completed correctly, spatial and temporal correspondences are maintained, or task constraints are adequately verified. Despite qualitative trajectory analyses, systematic evaluation of procedural correctness across the test set is lacking. Consequently, it remains difficult to distinguish failures in perception, skill selection, workflow organization, tool use, and answer mapping, or to determine whether correct answers result from valid and sufficiently verified procedures. The evaluation therefore provides limited evidence linking answer accuracy to domain-skill application and remote-sensing task execution capabilities.
Questions for authors
  1. What motivates the 13-task, five-dimension taxonomy? How are Measurement and Spatial Reasoning, Composite Reasoning, and Integrated Assessment distinguished?
  2. What are the 10 experts’ relevant backgrounds and review procedures? Please clarify reviewer assignments, disagreement resolution, and annotation agreement.
  3. Could GeoSkillAgent results with both Qwen models be provided, distinguishing transfer of the same evolved SkillBank from model-specific evolution?
  4. Why does GeoSkillAgent achieve 46.28% in Table 3 but 46.53% in Table 2? Were the same evaluation settings used?
  5. Why does Table 2 use an unweighted average of task accuracies rather than a weighted average based on the number of questions in each task?
  6. Please check lines 977–994 for wording or formatting issues.
Reviewer_tFbBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents a GeoSkillBench which contains 1385 evaluation questions for 13 UHR tasks. This paper also presents a GeoKillAgent which builds a library from failed trajectories on a dedicated training set to iteratively refine an expert-curated Skillbank. Candidate updates and the merged SkillBank are adopted only after they pass the validation accuracy gain and overall stability checks.

Critical strengths

A)-The authors proposed a GeoSkillBench to evaluate the skills.

B)- Authors adopted a scheme to update the skillbank while keep the models, tools, prompts and budget fixed.

C)-Authors presents a comparative study of how useful skills are across 13 tasks to show where expert guidance and self-generated plans differ, and the role of failure feedback in skill refinement.

D)- Authors provide a GeoSkillAgent which refines the skills via exposing the failure analysis with validity and stability checks.

Critical weaknesses

A) The skill improving is not a new and there exists many works on it. The main contribution lies in applying to remote sensing and it does not clear benefit why to apply for UHR? For example: EvoSkill

B) My major concern is that paper is claiming to provide validated skills but never shows the performance of diverse UHR benchmarks such as RSHR-Bench or XLRS-Bench. or this approach might give better on GEO-Bench-VLM. For a fair comparison, baseline model should have similar sizes.

C) In the introduction, author mentioned, “With GPT-5.6-Terra, paired refinement raises accuracy from 38.97% to 46.53%, a gain of 7.56 percentage”. But the table 2 accuracy does not have 38.97. how it is computed??

D) Authors claim every skill brings positive gain. But Table 2, shows that NS also beat the WS for example for T1.2, T2.4, T4.1, for GPT-5.6-Terra. Also, it is not clear that GeoSkillAgent (S1) which model it represents?

Questions for authors

A)- A clear motivation why it is required to adopt GeoSkillAgent for UHR? What if benchmark has diverse resolutions, should it also work?

B)- With clear similar baseline results and GeoSkillAgent over RSHR-Bench, XLRS-Bench, GEO-Bench-VLM to see whether skills benefit to get better scores?

C)- Why the skills are not bringing benefits against no skills from table 2?

Reviewer_GNCXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, to support the evaluation of Ultra-high-resolution (UHR) remote-sensing interpretation, the authors propose a new benchmark, GeoSkillBench. GeoSkillBench covers fine-grained skill annotations and a structured skill library supported by 1,385 evaluation questions across 13 UHR tasks and a shared geospatial tool environment. The authors also provide GeoSkillAgent, which is specifically constructed for this setting based on iterative refinement. Using this dataset, the authors evaluated GPT-5.6-Terra, Qwen3.5-9B, and Qwen2.5-VL-72B. The evaluation results show that expert-curated skills improve accuracy across all three models and the effectiveness of using GeoSkillAgent.

Critical strengths
  • To support the evaluation of Ultra-high-resolution (UHR) remote-sensing interpretation, this work proposes a new benchmark, GeoSkillBench.
  • GeoSkillBench covers fine-grained skill annotations and a structured skill library.
  • GeoSkillBench further comprises 1,385 evaluation questions across 13 UHR tasks and a shared geospatial tool environment.
  • The authors also propose GeoSkillAgent, a specifically constructed method for this setting.
  • Evaluation on GeoSkillBench is based on three different vision-language models: GPT-5.6-Terra, Qwen3.5-9B, and Qwen2.5-VL-72B.
  • Evaluation results show the importance of expert skills and the configuration to elicit their potential.
  • The ablation study on GeoSkillAgent also shows the importance of the expert library.
Critical weaknesses
  • The proposed benchmark, GeoSkillBench, is based on the conventional datasets XLRS-Bench and OmniEarth-Bench. This limits the contribution of this work.
  • Even though the work targets Ultra-high-resolution (UHR) remote sensing, the maximum resolutions for each vision encoder are not mentioned or discussed.
  • The reason why the authors chose these VLMs in the evaluation is not explained.
  • The agreement scores between 10 experts are not reported.
  • Even though the authors calculated 95% paired bootstrap intervals for each score, these are not mentioned or discussed in the main part of the paper.
Questions for authors

Please see Critical Weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.