- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces PolarTools, a benchmark for polar remote sensing. PolarTools can be regarded as a polar-specific version of general tool-augmented remote sensing image understanding, covering tasks such as iceberg, sea-ice, and polar oil-spill analysis. The benchmark also provides two variants depending on tool availability. The paper demonstrated the benchmarking of 13 models using PolarTools.
However, I believe that the paper lacks fundamental scientific writing standards and conventions. The main text is excessively verbose, and even essential information, such as how the dataset was constructed, is not sufficiently communicated. I believe that the paper requires substantial revisions from the ground up, which cannot be adequately addressed during the rebuttal period.
Critical strengths
This paper introduces PolarTools, a polar-specific benchmark for tool-augmented remote sensing image understanding. I believe that constructing a compiled dataset for polar remote sensing is a promising idea. I also find the idea of providing two evaluation modes, General Solving (GS) and Tool-Augmented Solving (TA), interesting. However, I have concerns about the reliability of the benchmark itself, particularly its construction process, as well as the clarity of the paper. Therefore, while the underlying concept has some strengths, I believe that the strengths of this paper are very limited.
Critical weaknesses
- Unclear motivation, practical value, and limited novelty.
- I understand the importance of polar remote sensing and the motivation for applying MLLMs to this domain. However, it remains unclear why a unified benchmark covering diverse domain-specific tasks is necessary. In practical applications, individual tasks are typically addressed using specialized models and task-specific evaluations, as demonstrated by AutoICE. What practical insights can be gained by evaluating general-purpose MLLMs across these tasks? Simply concluding that "current models remain unreliable in multi-step polar analysis" is insufficient. I would expect further experiments or discussions on how these models could be improved or adapted for real-world applications.
- PolarTools appears to primarily compile existing polar remote sensing datasets and tasks while adding tool-calling capabilities. As Figure 3 suggests, the benchmark largely appears to be a compilation of existing resources. It remains unclear what new insights this compilation provides. More fundamentally, the paper seems to conflate two different paradigms: task-specific applications of MLLMs and the evaluation of their general capabilities. Further clarification of this distinction and the benchmark's intended contribution is necessary.
- Insufficient explanation of benchmark construction.
- Although benchmark construction is the primary contribution, essential details such as data sources, annotation procedures, question generation, and quality verification are largely relegated to the appendix. Figure 3 appears to illustrate the construction process, but it is not adequately discussed in the main text. The manuscript contains many general explanations and examples, while the technical details necessary to assess the benchmark's reliability remain unclear.
- The main text is excessively verbose but lacks substantive methodological details. I have not reviewed the appendix, as I believe the main text should be sufficiently self-contained. The appendix should provide supplementary information rather than serve as additional pages for essential methodological explanations. The main text is not a lay summary and should contain sufficient scientific and technical detail to allow readers and reviewers to assess the validity of the proposed benchmark without relying on the appendix.
- Concerns about experimental validity and presentation.
- Line 369 explicitly states that "Results are provisional," which is difficult to understand in a submitted manuscript. Furthermore, the evaluation does not use a common question subset across models, and claims regarding execution traces and process-level evaluation are not sufficiently supported by quantitative results. These issues make it difficult to assess the reliability of the reported findings.
- Poor clarity and scientific writing.
- The manuscript contains substantial redundancy and unclear explanations, while important methodological details are difficult to identify. There are also inconsistencies and missing citations for the evaluated models. I strongly encourage the authors to substantially revise the manuscript, particularly its structure and technical presentation.
Questions for authors
- What practical value does a unified benchmark provide to the polar remote sensing community beyond existing task-specific evaluations? Have the authors considered task-specific adaptation or tuning to address the identified limitations?
- What is the fundamental contribution of PolarTools beyond compiling existing datasets and adding tool-calling capabilities?
- Why are essential details of benchmark construction largely relegated to the appendix rather than explained in the main text?
- What does "Results are provisional" in Line 369 mean? Why are provisional results presented as the main findings of a submitted manuscript?
- Where are the quantitative results supporting the claims regarding execution traces and process-level evaluation?
- Have the authors carefully verified its clarity, consistency, and citations, considering the common scientific manner?