acceptodds
Under review as a conference paper at ICLR 2027

PolarTools: Benchmarking Tool-Augmented Multimodal Agents for Polar Remote Sensing Analysis

Abstract

Recent advances in multimodal large language models (MLLMs) and autonomous agents have opened up new opportunities for automated scientific analysis. However, their capabilities in specialized domains such as polar remote sensing remain largely unexplored. Existing polar benchmarks primarily evaluate perception models, while it remains unclear whether domain-specific tools provide additional value beyond the growing code-generation capabilities of MLLMs. To address this gap, we introduce PolarTools, a benchmark for evaluating tool-augmented multimodal agents in polar remote sensing. PolarTools contains 710 question-answer instances covering iceberg, sea-ice, and polar oil-spill analysis, spanning perception and evidence interpretation; object and spatial structure analysis; temporal change and motion; geometric measurement and counting; quantitative product diagnostics; and preprocessing and cartography. We further develop a unified environment integrating 113 domain-specific tools for preprocessing, target extraction, geospatial analysis, temporal modeling, and visualization. Using PolarTools, we design two evaluation modes: General Solving (GS), in which agents are provided with a Python environment in a Docker container and rely on their own capabilities to solve tasks, and Tool-Augmented Solving (TA), in which agents additionally have access to polar-specific tools. We conduct experiments on 13 multimodal model configurations to analyze how domain-specific tools affect task completion, execution behavior, and efficiency. Unlike trajectory-matching evaluations, PolarTools evaluates agents based on task requirements and verifiable outputs, allowing diverse valid solutions while enabling analysis of whether domain-specific tools improve task completion and efficiency. PolarTools provides a testbed for studying multimodal agents in polar scientific workflows and the value of domain-specific tool augmentation.

Primary area:
datasets and benchmarks
Keywords:
Polar Remote Sensing, Multimodal Agents, Tool Use, Agent Evaluation, Stage-wise Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_RNQTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces PolarTools, a benchmark for polar remote sensing. PolarTools can be regarded as a polar-specific version of general tool-augmented remote sensing image understanding, covering tasks such as iceberg, sea-ice, and polar oil-spill analysis. The benchmark also provides two variants depending on tool availability. The paper demonstrated the benchmarking of 13 models using PolarTools.

However, I believe that the paper lacks fundamental scientific writing standards and conventions. The main text is excessively verbose, and even essential information, such as how the dataset was constructed, is not sufficiently communicated. I believe that the paper requires substantial revisions from the ground up, which cannot be adequately addressed during the rebuttal period.

Critical strengths

This paper introduces PolarTools, a polar-specific benchmark for tool-augmented remote sensing image understanding. I believe that constructing a compiled dataset for polar remote sensing is a promising idea. I also find the idea of providing two evaluation modes, General Solving (GS) and Tool-Augmented Solving (TA), interesting. However, I have concerns about the reliability of the benchmark itself, particularly its construction process, as well as the clarity of the paper. Therefore, while the underlying concept has some strengths, I believe that the strengths of this paper are very limited.

Critical weaknesses
  1. Unclear motivation, practical value, and limited novelty.
    • I understand the importance of polar remote sensing and the motivation for applying MLLMs to this domain. However, it remains unclear why a unified benchmark covering diverse domain-specific tasks is necessary. In practical applications, individual tasks are typically addressed using specialized models and task-specific evaluations, as demonstrated by AutoICE. What practical insights can be gained by evaluating general-purpose MLLMs across these tasks? Simply concluding that "current models remain unreliable in multi-step polar analysis" is insufficient. I would expect further experiments or discussions on how these models could be improved or adapted for real-world applications.
    • PolarTools appears to primarily compile existing polar remote sensing datasets and tasks while adding tool-calling capabilities. As Figure 3 suggests, the benchmark largely appears to be a compilation of existing resources. It remains unclear what new insights this compilation provides. More fundamentally, the paper seems to conflate two different paradigms: task-specific applications of MLLMs and the evaluation of their general capabilities. Further clarification of this distinction and the benchmark's intended contribution is necessary.
  2. Insufficient explanation of benchmark construction.
    • Although benchmark construction is the primary contribution, essential details such as data sources, annotation procedures, question generation, and quality verification are largely relegated to the appendix. Figure 3 appears to illustrate the construction process, but it is not adequately discussed in the main text. The manuscript contains many general explanations and examples, while the technical details necessary to assess the benchmark's reliability remain unclear.
    • The main text is excessively verbose but lacks substantive methodological details. I have not reviewed the appendix, as I believe the main text should be sufficiently self-contained. The appendix should provide supplementary information rather than serve as additional pages for essential methodological explanations. The main text is not a lay summary and should contain sufficient scientific and technical detail to allow readers and reviewers to assess the validity of the proposed benchmark without relying on the appendix.
  3. Concerns about experimental validity and presentation.
    • Line 369 explicitly states that "Results are provisional," which is difficult to understand in a submitted manuscript. Furthermore, the evaluation does not use a common question subset across models, and claims regarding execution traces and process-level evaluation are not sufficiently supported by quantitative results. These issues make it difficult to assess the reliability of the reported findings.
  4. Poor clarity and scientific writing.
    • The manuscript contains substantial redundancy and unclear explanations, while important methodological details are difficult to identify. There are also inconsistencies and missing citations for the evaluated models. I strongly encourage the authors to substantially revise the manuscript, particularly its structure and technical presentation.
Questions for authors
  1. What practical value does a unified benchmark provide to the polar remote sensing community beyond existing task-specific evaluations? Have the authors considered task-specific adaptation or tuning to address the identified limitations?
  2. What is the fundamental contribution of PolarTools beyond compiling existing datasets and adding tool-calling capabilities?
  3. Why are essential details of benchmark construction largely relegated to the appendix rather than explained in the main text?
  4. What does "Results are provisional" in Line 369 mean? Why are provisional results presented as the main findings of a submitted manuscript?
  5. Where are the quantitative results supporting the claims regarding execution traces and process-level evaluation?
  6. Have the authors carefully verified its clarity, consistency, and citations, considering the common scientific manner?
Reviewer_Y8FROpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces PolarTools, a benchmark for studying whether domain-specific tools provide additional value beyond the code-generation capabilities of MLLMs. The authors construct a benchmark covering diverse polar remote sensing tasks and compare general solving with tool-augmented solving. The results show that specialized tools generally improve task accuracy.

Critical strengths

The paper covers diverse polar remote sensing tasks that require multiple scientific operations, going beyond traditional visual prediction and question answering. It evaluates 13 model configurations under both general solving and tool-augmented solving, showing that domain-specific tools generally improve task accuracy.

Critical weaknesses
  1. Some existing remote sensing benchmarks already support multimodal inputs and complex tasks. Although the paper compares PolarTools with existing benchmarks, the need for a new benchmark is not fully clear, as existing benchmarks could potentially be extended to support the proposed evaluation.

  2. Table 2 shows large performance differences across models, such as GPT-6-Sol and GPT-4o. It is unclear whether these differences mainly come from the models' own reasoning abilities or their ability to understand and correctly use specialized tools. Moreover, some models perform worse with tool augmentation, but the reasons for these performance drops are not sufficiently analyzed. The current results provide limited insight into actual tool-use behavior.

Questions for authors
  1. Could the authors clarify why existing remote sensing benchmarks cannot be extended to support the proposed toolchain and process-aware evaluation, rather than constructing a new dataset?

  2. It would be helpful to provide more detailed tool-use statistics, such as tool invocation rates, tool selection accuracy, and execution success rates, to better understand the sources of performance improvements and the reasons for performance drops in some models.

  3. How are critical steps defined and evaluated across different task categories? In particular, how does the evaluation handle different valid workflows that lead to the same correct result?

Reviewer_kUaROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces PolarTools, a benchmark for multimodal agents performing polar remote sensing analysis, comprising 710 question–answer instances across six task families and an execution environment with 113 specialized tools. Tasks cover perception, spatial analysis, temporal change, geometric measurement, product diagnostics, and preprocessing. Evaluation uses task requirements and verifiable outputs rather than requiring a fixed tool trajectory. The authors evaluate 13 model configurations under General Solving (GS), which permits writing and executing Python, and Tool Augmented Solving (TA), which additionally provides access to PolarTools. They report final accuracy, intermediate scientific checkpoint attainment, execution failures, and timing. Tool benefits vary across models and task families, with larger gains in temporal analysis and preprocessing, but improvements are not universal. The appendix provides additional perception model experiments and ten illustrative case studies.

Critical strengths

The paper’s main strength is its substantial engineering contribution in building a unified tool and execution environment for polar remote sensing analysis. PolarTools organizes 113 specialized interfaces into five groups covering data access and preprocessing, perception and region extraction, measurement and statistics, temporal analysis, and output generation. It integrates model inference with deterministic scientific computations, connecting perception outputs to geographic measurements and temporal analysis while handling practical constraints such as coordinates, units, and valid pixels. This integration provides a useful foundation for studying how agents compose domain capabilities to complete polar analysis tasks

Critical weaknesses

W1: The benchmark’s added scientific value and construction reliability are insufficiently substantiated. Earth-Bench (Feng et al., 2026) already includes Greenland snow/ice and Arctic Ocean analysis tasks (questions 163–168 and 182–185), so PolarTools needs to establish its contribution beyond expanded task and tool coverage. Figure 1 emphasizes complex real scenarios without defining or characterizing task complexity. Several examples provide masks, polygons, or tracks and primarily require downstream computation, leaving unclear the coverage of autonomous problem decomposition, method selection, and analysis involving multiple stages. Section 4.1 also insufficiently explains independent validation of reference answers and scoring checks, and acknowledges incomplete verification of training/evaluation source separation across the benchmark. The stated anonymous repositories have no accessible links, and no supplementary attachments are provided. These gaps limit assessment of the benchmark’s distinctive contribution and construction reliability.

W2: The experiments do not sufficiently disentangle or explain tool benefits over general programming. Section 4.2 excludes the specialist package and its private weights from GS, whereas TA can invoke domain perception models. The comparison therefore measures the complete toolkit’s effect without separating contributions from additional perception capabilities, preimplemented scientific computations, and tool interfaces. Variation in gains across models and task families also lacks systematic explanation: Appendix E provides selected cases, and Failure-Rate measures execution errors; neither establishes the distribution of scientific errors. Results further rely on a single execution framework, leaving unclear whether GS fully exercises its coding and repair capabilities and whether tool gains depend on that framework. This limits the explanation of when and why tools help.

Reference

Feng, P., et al. (2026). Earth-Agent: Unlocking the Full Landscape of Earth Observation with Agents. https://arxiv.org/abs/2509.23141

Questions for authors
  1. Added capabilities and complexity (W1): Which tasks originate from actual scientific needs and require autonomous problem decomposition and method selection rather than prescribed computations? What proportion of the benchmark do they constitute, and how do they substantively differ from existing Earth-Bench tasks?

  2. Annotation expertise and review (W1): How many people participated in question design, reference construction, and review, and what expertise do they have in polar science or remote sensing? Was independent expert review conducted, how were disagreements resolved, and were programmatically expanded questions reviewed similarly?

  3. Systematic error analysis (W2): Have perception, tool selection, argument, coding, and scientific computation errors been quantitatively analyzed in paired GS/TA runs? Which errors decrease or increase with tool access, and do these changes explain gains across models and task families? Process analyses in Earth-Agent (Feng et al., 2026) and DORA (Wang et al., 2026), together with the failure attribution approach of Zhang et al. (2025), offer useful methodological references.

  4. Perception resources (W2): Which perception weights were actually used, and which tasks depended on them? Is there controlled evidence or analysis establishing how much benefit the complete toolkit retains when GS can access the same underlying perception models?

  5. Harness dependence (W2): How does GS support execution feedback, debugging, and iterative repair? Do results from another mature coding harness support the main findings under comparable models, tasks, perception resources, and budgets?

References

Reviewer_maJWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tests whether multimodal agents can carry out polar remote-sensing analysis, and whether specialized tools help beyond letting them write their own code. It introduces 710 tasks and 113 tools, then compares models with and without access to those tools. The tools often improve tasks involving scientific processing and measurement, but they don’t consistently help visual recognition or make solving faster.

Critical strengths

Its main strength is comparing specialized tools against a realistic baseline where models can write and run their own code. It also checks both final results and key scientific steps, helping reveal errors that a plausible-looking answer could hide.

Critical weaknesses
  • The biggest concern is that the paper hasn’t fully checked whether the perception tools were trained on images related to the benchmark. If a tool has already seen the same scene or a very similar image, better results may partly reflect familiarity rather than generalization. How much of the tool benefit remains on clearly held-out data?

  • Different models were evaluated on different question subsets, and the results are still provisional. Comparing a model with and without tools is more reasonable, but comparing models against each other is less reliable because some may have completed easier questions. A shared evaluation set would make the rankings much more convincing.

  • The comparison shows whether the whole toolkit helps, but not which part provides the benefit. The toolkit includes pretrained perception models, scientific algorithms, and convenient wrappers. Is the improvement coming from better visual models, access to specialized methods, or simply avoiding the need to write code? Separating those contributions would make the findings more useful.

  • The scientific evaluation needs closer scrutiny. The paper says it checks final outputs and critical steps, but those checks need to accept valid self-written solutions without favoring the provided tools. Can a different but scientifically correct method receive full credit, and how are ambiguous references or numerical tolerances handled?

  • The efficiency claim is quite limited. Time savings are measured only on questions that both modes answer correctly, so they don’t capture the full cost of failures, retries, or newly solved tasks. Reporting overall runtime, model usage, and tool-compute costs would give a clearer picture of whether the toolkit is practical.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.