- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper evaluates reusable domain skills for tool-augmented multimodal agents in ultra-high-resolution remote-sensing interpretation. The authors introduce GeoSkillBench, comprising 1,385 questions across 13 tasks, skill annotations, a structured skill library, and shared geospatial tools. With models, tools, and interaction budgets held fixed, it compares agents without skill documents, with expert-curated skills, and with self-generated plans through paired outcomes and execution traces. The authors also propose GeoSkillAgent, which extracts procedural corrections from failed training trajectories to refine the skill library while keeping model parameters fixed. The experimental analysis indicates that the benefits of skill guidance depend on both the task and the executing model. Expert-curated skills do not consistently outperform self-generated plans, and aggregate gains are accompanied by regressions on some questions. The authors further report that refining the skill library using failure feedback improves overall accuracy under the evaluated settings.
Critical strengths
The authors provide a structured library of domain-specific skill documents and an accompanying evaluation dataset, with clearly defined task and skill categories and quality control combining automated screening with expert review. Beyond providing performance benchmarks, more analyses offer empirical insights into when domain skills help and when they may lead to regressions. Furthermore, the authors introduce a skill-library refinement process driven by failed execution trajectories and incorporating validation and stability checks. Under the reported experimental settings, this process achieves higher overall accuracy than the static expert-curated library, offering a useful direction for future research on skill-guided remote-sensing agents.
Critical weaknesses
- Insufficient detail on dataset construction and skill evolution limits quality assessment and reproducibility. Although the paper describes its data sources, automated screening, and 13 task categories, the rationale for this taxonomy, boundaries between overlapping categories, and assignment of questions requiring multiple capabilities remain insufficiently explained. The reported review by 10 experts also lacks details on their expertise, responsibilities, reviewers per question, and agreement on answers and skill annotations. These omissions make the reliability of human quality control difficult to assess. For GeoSkillAgent, despite the reported partition sizes and scene-level separation, the training pool’s composition by source, task, and construction route, together with its complete selection procedure, remains unclear. This limits independent verification of data quality and reproduction of the refinement process and results.
- The benchmark primarily measures four-option multiple-choice accuracy, supplemented by skill access, tool errors, interaction counts, and budget exhaustion. These statistics capture some execution problems but do not directly assess whether critical steps are completed correctly, spatial and temporal correspondences are maintained, or task constraints are adequately verified. Despite qualitative trajectory analyses, systematic evaluation of procedural correctness across the test set is lacking. Consequently, it remains difficult to distinguish failures in perception, skill selection, workflow organization, tool use, and answer mapping, or to determine whether correct answers result from valid and sufficiently verified procedures. The evaluation therefore provides limited evidence linking answer accuracy to domain-skill application and remote-sensing task execution capabilities.
Questions for authors
- What motivates the 13-task, five-dimension taxonomy? How are Measurement and Spatial Reasoning, Composite Reasoning, and Integrated Assessment distinguished?
- What are the 10 experts’ relevant backgrounds and review procedures? Please clarify reviewer assignments, disagreement resolution, and annotation agreement.
- Could GeoSkillAgent results with both Qwen models be provided, distinguishing transfer of the same evolved SkillBank from model-specific evolution?
- Why does GeoSkillAgent achieve 46.28% in Table 3 but 46.53% in Table 2? Were the same evaluation settings used?
- Why does Table 2 use an unweighted average of task accuracies rather than a weighted average based on the number of questions in each task?
- Please check lines 977–994 for wording or formatting issues.