- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SeaIce-Bench, an evaluation suite covering sea-ice concentration estimation from satellite imagery, numerical forecasting of sea-ice extent and regional motion, and literature-derived question answering. The experiments evaluate comprehensive MLLM API models and domain-specific small models. The paper also discusses the performance differences across these tasks.
Critical strengths
- The proposed benchmark is comprehensive, examining three important capabilities in sea-ice analysis: remote sensing image understanding, numerical prediction, and textual QA. The scale is decent: 136,500 multispectral satellite images, monthly sea-ice extent records and weekly sea-ice motion vectors spanning years, and 53,785 question-answer pairs derived from the literature.
- The evaluation is comprehensive. The paper includes zero-shot evaluation of 10 frontier API models and domain-specialized small models, giving a full picture of current MLLM performance on sea-ice analysis and how it compares with domain-specialized tools.
Critical weaknesses
The paper states that sea-ice analysis requires models to connect remote sensing images, geophysical records, and specialized scientific knowledge. The paper therefore proposes SeaIce-Bench, combining tasks from these three sources. However, the tasks are separate within their own genres and are not associated with one another. It would be interesting to construct some long-horizon scientific tasks where the model needs to synergize visual understanding from satellite images, textual QA, and numerical prediction, and to report end-to-end performance and per-stage diagnostics. For the current draft, if time permits, adding some qualitative results would better justify the benchmarking of the three capabilities and promote the value of the paper.
Questions for authors
- The expert review of sampled QA pairs is a valuable quality-control step. Could the paper report more details, such as the review criteria, the proportion of items corrected or removed, the agreement rate among experts, and the main issues identified?
- The literature section should discuss similar MLLM benchmarking efforts across other Earth science or geoscience subdomains to better present the current progress in this direction, position the proposed SeaIce-Bench as filling the gap, and illustrate the need for it.
We thank the reviewer for the constructive questions. Both led to changes in the manuscript, which we describe below.
Q1. Details of the expert review of the QA pairs
We have rewritten Appendix M ("Expert Verification of the Textual QA Corpus") to report the curation process in full. The main points are as follows.
Proportion removed. GPT-5.5 generated approximately 60,000 candidate QA pairs. Curation removed approximately 6,200 of them (about 10%), which leaves the 53,785 pairs in the released corpus.
Two-stage process.
Review criteria. An expert retained an item only if its reference answer could be verified against the source text from which the item was generated and, for a calculation item, only if the calculation had been checked independently. In total, 5,770 pairs passed expert verification (10.7% of the corpus; the breakdown by task family is in Table 7). Appendix N.5 shows examples of the outcome: each item is listed with the verification notes and, where available, the source used to verify it.
Main issues identified. The removed items fall into three categories:
What we cannot report. We did not record how many items were removed in each category or for each question type. We therefore report only the overall removal figure in the revised manuscript.
Q2. MLLM benchmarks in other Earth-science and geoscience subdomains
We revised "MLLM benchmarks in other Earth-science subdomains," to Section 2 (Related Work), with eight new references:
How this positions SeaIce-Bench. These benchmarks generally find that general-purpose MLLMs fall short on domain-specific tasks, which is consistent with our visual and motion-vector results. Sea ice, however, is either absent from them or a small part of a broad suite. To our knowledge, none of them combines native multispectral sea-ice imagery, multi-decadal extent and motion records, and sea-ice literature. This is the gap that SeaIce-Bench fills, and we have updated the "Positioning of SeaIce-Bench" paragraph in Section 2 to say so.