acceptodds
Under review as a conference paper at ICLR 2027

SEAICE-BENCH: A Benchmark for Sea Ice Prediction and Analysis Capability of LLMs

Abstract

Sea ice analysis requires models to connect remote-sensing imagery, long-horizon geophysical records, and specialized scientific knowledge, yet existing large language model (LLM) benchmarks do not evaluate this combination of capabilities in a sea-ice-specific setting. We introduce SEAICE-BENCH, a benchmark for assessing LLMs and multimodal LLMs on three evidence sources: native Sentinel-2 imagery, historical numerical records, and domain literature. SEAICE-BENCH comprises 136,500 multispectral satellite images across full-scene and crop-based splits, monthly sea ice extent records from November 1978 to January 2026, weekly sea ice motion vectors from November 1978 to December 2024, and a curated corpus of more than 10,000 sea-ice research articles and textbooks that yields 53,785 question-answer pairs spanning six task families and four answer formats. We evaluate ten frontier models, seven proprietary and three open-weight, in a unified zero-shot protocol on sea ice concentration estimation, next-step sea ice extent and motion vector reasoning, and literature-grounded question answering. The results reveal substantial headroom for current systems: models remain weak at processing multiple large image files to estimate sea ice concentration in specific regions and at predicting multidimensional motion vectors. These findings position SEAICE-BENCH as a realistic testbed for measuring progress toward expert-facing LLM systems for sea-ice analysis, forecasting support, and scientific literature synthesis.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, LLMs, Sea Ice, AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9cH1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SeaIce-Bench, an evaluation suite covering sea-ice concentration estimation from satellite imagery, numerical forecasting of sea-ice extent and regional motion, and literature-derived question answering. The experiments evaluate comprehensive MLLM API models and domain-specific small models. The paper also discusses the performance differences across these tasks.

Critical strengths
  1. The proposed benchmark is comprehensive, examining three important capabilities in sea-ice analysis: remote sensing image understanding, numerical prediction, and textual QA. The scale is decent: 136,500 multispectral satellite images, monthly sea-ice extent records and weekly sea-ice motion vectors spanning years, and 53,785 question-answer pairs derived from the literature.
  2. The evaluation is comprehensive. The paper includes zero-shot evaluation of 10 frontier API models and domain-specialized small models, giving a full picture of current MLLM performance on sea-ice analysis and how it compares with domain-specialized tools.
Critical weaknesses

The paper states that sea-ice analysis requires models to connect remote sensing images, geophysical records, and specialized scientific knowledge. The paper therefore proposes SeaIce-Bench, combining tasks from these three sources. However, the tasks are separate within their own genres and are not associated with one another. It would be interesting to construct some long-horizon scientific tasks where the model needs to synergize visual understanding from satellite images, textual QA, and numerical prediction, and to report end-to-end performance and per-stage diagnostics. For the current draft, if time permits, adding some qualitative results would better justify the benchmarking of the three capabilities and promote the value of the paper.

Questions for authors
  1. The expert review of sampled QA pairs is a valuable quality-control step. Could the paper report more details, such as the review criteria, the proportion of items corrected or removed, the agreement rate among experts, and the main issues identified?
  2. The literature section should discuss similar MLLM benchmarking efforts across other Earth science or geoscience subdomains to better present the current progress in this direction, position the proposed SeaIce-Bench as filling the gap, and illustrate the need for it.
1 comment

We thank the reviewer for the constructive questions. Both led to changes in the manuscript, which we describe below.

Q1. Details of the expert review of the QA pairs

We have rewritten Appendix M ("Expert Verification of the Textual QA Corpus") to report the curation process in full. The main points are as follows.

Proportion removed. GPT-5.5 generated approximately 60,000 candidate QA pairs. Curation removed approximately 6,200 of them (about 10%), which leaves the 53,785 pairs in the released corpus.

Two-stage process.

  1. Automatic screening. A screening pass with GPT-5.5 removed candidates that duplicated another item, did not follow the required answer format, or leaked the correct answer in the question stem. Format conformance was also enforced by programmatic validation. For a multiple-choice item, for example, the validator requires that all required fields are present, that the answer key is one of A, B, C, or D, and that the option text for that key exists.
  2. Expert review. Sea-ice experts then examined sampled items and removed those that they could not verify against the source.

Review criteria. An expert retained an item only if its reference answer could be verified against the source text from which the item was generated and, for a calculation item, only if the calculation had been checked independently. In total, 5,770 pairs passed expert verification (10.7% of the corpus; the breakdown by task family is in Table 7). Appendix N.5 shows examples of the outcome: each item is listed with the verification notes and, where available, the source used to verify it.

Main issues identified. The removed items fall into three categories:

  • the question stem leaked the correct answer;
  • the generated question or answer was not in the required format;
  • an expert could not verify the content against the source.

What we cannot report. We did not record how many items were removed in each category or for each question type. We therefore report only the overall removal figure in the revised manuscript.

Q2. MLLM benchmarks in other Earth-science and geoscience subdomains

We revised "MLLM benchmarks in other Earth-science subdomains," to Section 2 (Related Work), with eight new references:

  • Earth observation. Zhang and Wang (CVPR Workshops 2024) benchmark GPT-4V on scene understanding, localization, counting, and change detection. VRSBench (Li et al., NeurIPS 2024) pairs 29,614 remote-sensing images with captioning, grounding, and visual question answering tasks. GEOBench-VLM (Danish et al., ICCV 2025) covers 31 fine-grained geospatial tasks.
  • Meteorology. WeatherQA (Ma et al., 2024) tests reasoning about severe convective weather on more than 8,000 image–text pairs. ClimateIQA (Chen et al., 2024) poses visual questions about extreme-weather events on wind-gust heatmaps.
  • Geology. GeoMap-Bench (Huang et al., CVPR 2025) evaluates geologic-map understanding.
  • Ocean science. OceanBench accompanies OceanGPT (Bi et al., ACL 2024) with 15 ocean-science tasks.
  • Cross-sphere. OmniEarth-Bench (Wang et al., 2025) spans six Earth spheres with 29,855 annotations built from observational data. Of these, 230 concern the cryosphere (sea-ice forecasting and glacier analysis).

How this positions SeaIce-Bench. These benchmarks generally find that general-purpose MLLMs fall short on domain-specific tasks, which is consistent with our visual and motion-vector results. Sea ice, however, is either absent from them or a small part of a broad suite. To our knowledge, none of them combines native multispectral sea-ice imagery, multi-decadal extent and motion records, and sea-ice literature. This is the gap that SeaIce-Bench fills, and we have updated the "Positioning of SeaIce-Bench" paragraph in Section 2 to say so.

Reviewer_ksdfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SEAICE-BENCH, aimed at evaluating the capabilities of large language models and multimodal models in sea ice analysis tasks. This benchmark consists of three parts. The first part is a visual task, using Sentinel-2 multispectral satellite images, including 6,500 raw scene images and 130,000 cropped images. The model is required to estimate sea ice concentration using true color images and 12 spectral bands. The second part is a numerical prediction task, using sea ice extent time series and sea ice motion vector data since 1978 to evaluate the model's ability to predict the sea ice extent for the next month and the regional motion field for the next week. The third part is scientific literature question answering, generating 53,785 Q&A samples from over 10,000 research papers and textbooks related to sea ice, covering six types of scientific knowledge and reasoning tasks. The authors evaluated ten large models and compared them with traditional methods such as MMSeaIce, EA-LSTM, Attention-LSTM, and Persistence. The main finding of the paper is that current LLMs have certain capabilities in text question answering and univariate sea ice extent prediction, but there are significant shortcomings in multispectral image interpretation and spatial motion field prediction.

Critical strengths

1: Integrating remote sensing vision, long-term geophysical time series, and domain scientific knowledge into a unified evaluation framework has practical application value. Existing multimodal benchmarks usually focus on natural image understanding or use charts from scientific papers to test reasoning abilities. SEAICE-BENCH directly uses raw satellite images and historical sea ice observation records, thus having a clearer scientific application background. Especially, the paper not only tests visual recognition but also distinguishes prediction tasks into one-dimensional sea ice extent prediction and high-dimensional sea ice motion field prediction. This helps to reveal the differences in the ability of different models to handle various types of scientific data.

2: Introducing traditional scientific models as a baseline makes the experimental conclusions more convincing than merely comparing different LLMs. Table 3 of the paper shows that in the sea ice concentration prediction task, the specially trained MMSeaIce significantly outperforms the general-purpose MLLM. For example, the WCP-RMSE on cropped images is 2.06 for MMSeaIce, while the best general-purpose MLLM is 34.14. On the other hand, Table 4 shows that certain LLMs can compete with EA-LSTM in monthly sea ice extent predictions, but perform significantly worse in full-field predictions. These results indicate that general models are not equally effective for all scientific tasks and also demonstrate that science-specific models still hold significant value.

Critical weaknesses

1: The reference labels for visual tasks come from spectral rules rather than fully independently verified actual sea ice concentrations, which may affect the scientific validity of the Benchmark. This is my main concern regarding the paper. The author constructs pixel-level snow and ice masks based on NDSI and near-infrared reflectance, and then calculates sea ice concentration. But the spectral identification results of snow and ice are not necessarily equivalent to the actual sea ice concentration. The snow cover, melt ponds, cloud layers, cloud shadows, thin ice, and varying lighting conditions on the sea ice surface may lead to significant classification errors. Therefore, the current evaluation of the Benchmark may not fully reflect the model's ability to estimate the actual sea ice concentration, but rather its ability to predict the labels generated by the spectral rules set by the authors. This issue is directly related to the most important visual experimental results of the paper. In particular, the authors trained the adapted MMSeaIce on the same task data. If the training labels and test labels come from the same set of rules, then the high accuracy of MMSeaIce may partly reflect its ability to learn the label generation rules, rather than necessarily proving that its estimates of the actual sea ice state have the same accuracy. Therefore, the paper needs to clearly distinguish between "the performance of predicting automatically generated labels" and "the performance of predicting actual sea ice concentration."

2: The current numerical prediction experiments have not yet fully eliminated the impact of data time division and output format on model ranking. The sea ice extent prediction uses approximately 40 years of historical records to forecast the next month, while the motion field prediction uses long-term historical fields to forecast the next week. But for scientific time series evaluation, two issues need to be particularly focused on. First, is the data strictly divided by time, ensuring that the observation records during the test period are not indirectly influenced by training, model pre-training, or context construction? Especially since general LLMs may have encountered some publicly available historical sea ice data during their pre-training. If there is an overlap between the historical period of the test data and the model's knowledge cutoff date, the performance may include contributions from the memory of historical data. Secondly, the paper uses WCP-RMSE, assigning the most severe penalty to results that do not comply with the output format. This metric can reasonably evaluate task completion ability, but it also mixes prediction error with instruction adherence ability. For example, the model might generate a nearly correct motion field, but receive a poor score because the CSV output format does not meet the requirements. Therefore, the conclusion in the paper that "the model cannot perform high-dimensional geophysical predictions" should further distinguish between insufficient scientific prediction capability and insufficient ability to generate long-structured results.

Questions for authors

Question 1: Have the sea ice concentration reference labels undergone independent scientific validation? Can the authors provide a set of test samples labeled by sea ice experts or compared with authoritative sea ice concentration products? It is recommended to further report the errors between the NDSI-generated labels and the independent labels, and to analyze the errors based on cloud cover, sea ice type, season, and region.

If there is a significant bias in the labels themselves, how should the performance differences between the models be interpreted? Question 2: Can different sources of ground truth labels be used to validate the performance gap between MMSeaIce and MLLM? The advantage of the specialized model in the current Table 3 is very clear, but it is trained under the prediction target defined by the authors. If the evaluation target is replaced with independent sea ice concentration annotations, does the performance gap still hold? This is a key experiment to determine whether the benchmark has scientific explanatory value.

Question 3: Does the numerical prediction experiment strictly avoid time information leakage? Please clearly specify the time ranges for training, validation, and testing, and explain which historical data each model has access to. Especially for monthly sea ice extent predictions, it is recommended to use strict rolling time evaluation and report the results of seasonal baselines, historical mean values, or seasonal persistence prediction methods.

Question 4: Is it possible to separate prediction accuracy from output format errors? Currently, WCP-RMSE reflects two different types of failures simultaneously. It is recommended to report RMSE for valid outputs separately, while retaining the completion rate and WCP-RMSE. In addition, for the sports field output, further analysis can be conducted on directional error, speed magnitude error, and spatial consistency, rather than relying solely on a single comprehensive error.

Question 5: Why are the three data modalities evaluated separately, rather than providing a cross-modal fusion task? The main motivation of the paper emphasizes the comprehensive use of satellite imagery, numerical records, and scientific literature, but Section 4 clearly states that the three modules are evaluated independently. Could the author explain how this design supports the assessment of integrated sea ice science analysis capabilities? Even without building a complete end-to-end system, a small number of cases that require the simultaneous use of images and historical time series can be added to verify whether different information sources are truly complementary.

4 comments
AuthorsOpenReview ↗

Response to questions (part 1)

We thank the reviewer for the detailed questions. We answer them in order.

Q1. Independent validation of the sea-ice concentration labels

The reference follows the standard in our field. Deriving sea ice concentration from NDSI together with a near-infrared reflectance test is standard practice in sea-ice remote sensing, and we treat the resulting concentration as the ground truth. Appendix B gives the details.

Why the labels are not manual annotations. Pixel-by-pixel manual annotation is impractical at this scale. A single of one full scene is 10980 × 10980 pixels (Appendix G, Table 6), about 120 million pixels, and the benchmark contains 500 full-scene locations and 10,000 crop locations.

On label bias. As a team of sea-ice experts, we agree with the concentrations that the rule produces, and we do not see a significant bias in the labels. The same rule is applied to every scene and crop, in both hemispheres and in all seasons. Cloudy scenes are kept on purpose and pass through the same rule, because cloud cover is a routine condition in optical remote sensing.

Q2. Validating the gap between MMSeaIce and the MLLMs with other label sources

The comparison uses the same test set. Adapted MMSeaIce and all MLLMs are evaluated on the same test instances with the same reference labels. The baseline is trained only on the training split (Section 4).

Why we cannot use a different data source. Sentinel-2 is the best satellite source available to us for this task. Its label bands have 10 m to 60 m resolution. Commercial satellites with finer resolution are too expensive to use at this scale, and other public satellite sources do not reach a resolution of about 10 m. For example, the Sea Ice Index used in Section 3.2 is gridded at 25 km, while one of our crops is about 11 km across, so a crop is smaller than a single cell of that product. An independent label at comparable resolution is therefore not available.

Q3. Time-information leakage in the numerical experiments

The input never contains the target or later observations. Each extent instance gives the model the 480 months immediately before the target month and asks for the following month (Section 3.2). Each motion instance gives weekly fields before the target week and asks for the next week (Section 3.3). The LLMs are evaluated zero-shot and are not fine-tuned on any benchmark data (Section 4).

Exposure during pre-training. We are aware that historical sea-ice records are public and may be part of the pre-training data of the evaluated LLMs. We handle this in two ways:

  • Tool use, search, and code execution are disabled, so no model can look up a value at inference time (Section 4.1).
  • We extended the records to the latest available data, January 2026 for extent. The evaluation therefore includes recent observations that the models cannot have seen during pre-training for the used models.

If a model has internalised knowledge of sea-ice behaviour, such as the seasonal cycle and the long-term trend, we consider this acceptable to some extent. This knowledge is part of the domain competence that the benchmark is meant to measure.

The satellite images cannot have been seen. We processed every image in the visual component ourselves, and the processed images and their concentration labels did not exist before this work. No evaluated LLM had access to them.

Baselines. Table 4 already reports a persistence baseline next to the learned baselines.

AuthorsOpenReview ↗

Response to questions (part 2)

Q4. Separating prediction accuracy from output-format errors

RMSE over valid outputs. We will report RMSE over valid outputs only, next to completion rate and WCP-RMSE, both of which we keep.

Two parts of the current version already help to separate the two failure types. For every model with a completion rate of 100%, no penalty is applied, so its WCP-RMSE is the ordinary RMSE. Appendices I to K give qualitative error analyses that distinguish prediction errors from failures to process the inputs or to follow the output format.

"Sports field output." Our manuscript does not contain a sports field output. SeaIce-Bench has three components: sea-ice concentration estimation from Sentinel-2 imagery, next-month extent and next-week motion-vector forecasting from historical records, and question answering built from sea-ice literature. None of the tasks, inputs, or outputs concerns a sports field, so this part of the question does not apply to our manuscript. If the reviewer had a different output in mind, we would be glad to respond once it is clarified.

Q5. Why the three modalities are evaluated separately

This is a deliberate design choice, stated in Sections 3.1 and 4 and in the Conclusion, where we say that this first release is not a complete end-to-end workflow. The three components measure observation, prediction, and scientific understanding, and we evaluate them separately for two reasons.

  • Attribution. When a model fails a fused task, it is not possible to tell which evidence source caused the failure. Separate tasks show that the same models are strong on literature QA and scalar extent forecasting and weak on multispectral concentration estimation and motion-vector prediction.
  • Comparability. Each component can be run on its own, so text-only LLMs, multimodal LLMs, and task-specific baselines can each be compared on the components they support (Section 3.1).

The results also indicate that fusion is not yet the limiting factor. The strongest MLLMs have errors above 34 percentage points on the image task alone, so a task that combines images with time series would be limited by this single-modality weakness. We regard cross-modal cases that combine imagery with historical records as a extension of the benchmark, which the Conclusion describes as an initial and extensible foundation. For now, we are currently working on a small number of cases that require the simultaneous use of images and historical time series for satisfying the request.

Thank you for the response. I will keep my score at 2.

A correction: "sports field output" in my Q4 was a wording error; I meant the motion-vector field.

My main concern on label validity remains. I asked for a validated subset, not full pixel-level annotation, and the response gives no quantitative check. Since MMSeaIce is trained and tested on labels from the same rule, Table 3 shows agreement with that rule rather than accuracy on actual sea ice concentration.

AuthorsOpenReview ↗

Clarification for the main concern

We thank the reviewer for the follow-up, and for clarifying that the request is for a validated subset.

1. The rule-based label is the ground truth used in this sub-domain

The label rule in Appendix B is not one we designed. It is the established rule for detecting snow and ice in optical satellite imagery, and classified optical imagery is what the sea-ice community itself uses as the reference.

  • The rule and its thresholds. The MODIS snow-mapping algorithm classifies a pixel as snow or ice when NDSI > 0.4 and the near-infrared reflectance is above 0.11 (Hall et al., 1995; Hall et al., 2002). Appendix B applies the same two tests with the same thresholds to the corresponding Sentinel-2 bands: B03 and B11 for NDSI, and B08 for the near-infrared test.
  • Operational sea-ice products. The NDSI approach is the basis of NASA's sea-ice extent products from MODIS and its successor VIIRS (Riggs et al., 1999).

In this sub-domain there is no measurement of sea-ice concentration at 10 m resolution that ranks above the classified optical image. In-situ observations do not cover the area of a scene, and the other satellite products are far coarser. This is why we treat the SIC produced by the rule as the ground truth.

2. What we do for quality control of the images

The quantitative check on a subset would require a ratio of the label of a scene. The number of ice pixels divided by the number of valid pixels (Eq. 3). An independent verified value of this ratio can only be obtained by classifying the pixels of that scene independently. There are two possible sources.

  • Another product. None exists at a comparable resolution, as we explained in our first response (Q2). The Sea Ice Index is gridded at 25 km, and our images are in the 10 - 60 m resolution, which is much higher resolution.
  • Manual labelling by experts. If the output of the rule is not accepted as ground truth, the only remaining reference is an expert deciding, pixel by pixel, whether each pixel is ice. For the result to be a concentration, this must be done for every valid pixel of every scene in the subset. A single image in one scene has about 1.2 million pixels in a 10 m band (1098 × 1098). It is impractical to do that by human experts.

Therefore, a validated subset in quantitative way is not an alternative to pixel-level annotation. Instead, we conduct qualitative check for the quality control.

The check we did perform is qualitative. The sea-ice experts on our team inspected the generated labels together with the corresponding images. We agree with the labels.

3. Adapted MMSeaIce and Table 3

As the label is the ground truth of this sub-domain as mentioned above, Table 3 would show the accuracy on SIC.

4. Changes to the manuscript

To our best knowledge, SeaIce-Bench is the first benchmark for LLMs in the sea-ice domain. The image data are also unique. We processed every image, and the processed images and their concentration labels did not exist before. The benchmark is the first of its kind, we expect many of its readers to come from outside sea-ice remote sensing, and practices that are standard in our sub-domain need to be explained and supported for them. Therefore, we will add the explanation above and the references below to the manuscript, in Section 3.4 and Appendix B, so that the basis of the rule-based label is stated in the paper itself. Also, we will state there the qualitative check conducted by our sea-ice experts.

References

  • Hall, D. K., Riggs, G. A., and Salomonson, V. V. (1995). Development of methods for mapping global snow cover using Moderate Resolution Imaging Spectroradiometer data. Remote Sensing of Environment, 54(2), 127–140.
  • Hall, D. K., Riggs, G. A., Salomonson, V. V., DiGirolamo, N. E., and Bayr, K. J. (2002). MODIS snow-cover products. Remote Sensing of Environment, 83(1–2), 181–194.
  • Riggs, G. A., Hall, D. K., and Ackerman, S. A. (1999). Sea ice extent and classification mapping with the Moderate Resolution Imaging Spectroradiometer Airborne Simulator. Remote Sensing of Environment, 68(2), 152–163.
Reviewer_JDw4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SEAICE-BENCH for evaluating large language models and multimodal large language models on sea-ice analysis and prediction. The benchmark consists of three independent but complementary components: sea-ice concentration estimation from Sentinel-2 multispectral imagery, next-month extent and next-week regional motion-vector forecasting from historical records, and question answering constructed from domain literature. The authors evaluate ten models and include specialized vision and time-series baselines. The results show that LLMs are competitive on textual QA and some one-step scalar forecasting tasks, while substantial limitations remain in multispectral image interpretation and regional motion-vector prediction.

Critical strengths

The benchmark covers practically relevant and complementary sea-ice tasks. It brings native remote-sensing imagery, long-term numerical records, and domain-literature QA into a common evaluation suite, rather than simply reformulating general QA in a sea-ice context. Comparing image estimation, scalar forecasting, and spatial vector forecasting helps distinguish different types of model capability.

Critical weaknesses

1、The independent validity of the visual reference labels is insufficiently established. Section 3.4 states that sea-ice concentration is derived from NDSI and near-infrared reflectance rules, while some cloudy scenes are retained. Table 3 therefore primarily measures agreement with these rule-generated targets. The paper does not provide sufficient evidence of how well the labels agree with independent manual annotations or a trusted reference product across different ice-surface and observation conditions. This distinction is particularly important because the specialized baseline is trained on the benchmark’s training data. Its low error demonstrates better prediction of the benchmark targets, but does not by itself establish equivalent accuracy against actual sea-ice concentration. I would appreciate a modest independent validation covering different observation conditions, together with clarification of how invalid pixels and cloud-obscured areas are handled.

2、The current metrics and input protocol do not sufficiently separate scientific prediction capability from interface and formatting capability. Section 5.1 treats responses containing explanatory text or alternative estimates as invalid and assigns them the maximum error penalty. This is meaningful for evaluating usability in a strict automated pipeline, but should not be equated entirely with scientific prediction error. For example, Kimi K2.6 completes only 0.54% of the visual instances in Table 3, so its WCP-RMSE close to 100 predominantly reflects task non-completion rather than the accuracy of valid predictions. Reporting completion rate alongside error is helpful, but matched-instance prediction errors and a limited sensitivity analysis for formatting deviations containing a single unambiguous answer would make the results more interpretable. Furthermore, Section 4.1 disables tools and code execution while requiring long-history processing and complete spatial-field outputs. The findings should therefore be interpreted specifically as performance under a tool-free protocol, rather than as a direct measure of practical scientific-assistant capability.

Questions for authors

Could the authors clarify whether the entire manuscript, including all figures, underwent a thorough review and quality check before submission? The paper contains numerous defensive statements that anticipate potential criticism, while substantive explanations of the methodology and experiments remain insufficient. There are also formatting issues in the contributions portion of Section 1. In addition, the presentation of the framework diagram raises concerns about whether it received adequate human verification. Please clarify the extent to which generative AI was used in preparing the text and figures, and describe the verification procedures followed. I strongly recommend thoroughly reviewing all AI-assisted content before submission to ensure its accuracy, clarity, and consistency with the research actually conducted.

8 comments
AuthorsOpenReview ↗

Response to the questions

We thank the reviewer for the time spent on our submission.

Q1. Was the entire manuscript, including all figures, reviewed and quality-checked before submission?

Yes. The authors reviewed all content before submission: main text, tables, figures, captions, and appendices. As stated in the manuscript, we take responsibility for the final content.

Q2. To what extent was generative AI used, and how was it verified?

Our use of generative AI is reported in the manuscript, in the "Generative AI Disclosure" section that follows the Conclusion. In summary:

  • Benchmark construction. GPT-5.5 generated candidate question–answer pairs from source passages for the textual component (Section 3.5). These were verified in three steps: rule-based checks removed malformed, duplicated, and answer-leaking items; semantic review removed unsupported or ambiguous items; and sea-ice experts manually reviewed 5,770 randomly sampled pairs, 10.7% of the corpus (Appendix M). GPT-5.5 is excluded from the evaluated models to avoid generator–evaluator overlap (Section 4.1).
  • Experimental subjects. The ten evaluated LLMs and MLLMs produced the outputs that the benchmark scores.
  • Writing. Generative AI tools were used for proofreading and typographical-error checking.
  • Not used. We did not use generative AI to develop the conceptual framework, propose hypotheses, design the methodology, or interpret results.

We reviewed all AI-assisted work.

Q3. "Defensive statements that anticipate potential criticism"

These statements were not written in anticipation of criticism. An earlier version of this paper was submitted to NeurIPS 2026, where it received borderline scores from four reviewers. We addressed every issue raised in those four reviews, and the statements the reviewer noticed are our answers to questions that those reviewers actually asked. One example is visible in the paper: Appendix N.2 notes that the human survey was added in response to a reviewer's request for a human reference for the texual questions.

We have not posted a preprint, and the NeurIPS submission and its reviews were not released publicly on OpenReview. We cannot share those records at this stage without violating the anonymity of double-blind review. To support the above claim, we commit to making the reviews of the NeurIPS 2026 submission public once this manuscript is accepted.

On the completeness of the methodology: data construction is described in Section 3 and the protocol in Section 4, with the details in the appendices: reference-label construction (B), QA construction prompts (D), evaluation prompts (F), the adapted baseline (G), metric definitions (H), error analyses for each task (I–K), expert verification (M), the human evaluation (N), and model versions (O). If the reviewer can name a specific step that remains unclear, we will expand it.

Q4. Formatting of the contributions in Section 1

There is no formatting issue in this part. The contributions are typeset with the standard LaTeX itemize environment (\begin{itemize} ... \end{itemize}) under the official ICLR 2027 style file, with no manual spacing or custom formatting. In the compiled PDF they appear as a regular three-item bulleted list at the top of page 2.

The submitted PDF was generated on Overleaf directly from this LaTeX source. Sharing the project now would violate anonymity, so we will upload the original LaTeX source files to arXiv once this manuscript is accepted, where they can be inspected.

AuthorsOpenReview ↗

Response to weaknesses

Weakness 1: Validity of the visual reference labels, and cloudy scenes

Cloudy scenes are retained on purpose. Section 3.4 states that we keep some cloudy acquisitions "to reflect realistic observation conditions." We are a team of sea-ice researchers, and this is a deliberate design decision, not an oversight. Sentinel-2 MSI is an optical sensor, so the surface is sometimes blocked by cloud. This is a routine condition in remote sensing of the polar regions. A benchmark restricted to cloud-free scenes would not represent the imagery that sea-ice analysts, or a model assisting them, actually receive.

The reference follows the standard in our field. Deriving ice cover from NDSI together with a near-infrared reflectance test is standard practice in optical sea-ice remote sensing, and we treat the resulting concentration as the ground truth. Appendix B gives the full details. SIC is the percentage of ice pixels among the valid pixels of the scene or crop (Eq. 3). Cloudy scenes go through the same rule: a pixel that does not pass both tests, whether it is open water or obscured by cloud, is not counted as ice. The rule is deterministic and fully specified.

Why the labels are not manual annotations. Pixel-by-pixel manual annotation is impractical at this scale. A single band of one full scene is 10980 × 10980 pixels (Appendix G, Table 6), about 120 million pixels, and the benchmark contains 500 full-scene locations and 10,000 crop locations. As a team of sea-ice experts, we agree with the concentrations that the standard produces.

Weakness 2: Prediction capability versus format compliance, and the tool-free protocol

Why we use a worst-case penalty. If RMSE is computed only over valid outputs, a model that fails to answer the difficult instances is scored only on the instances it chose to answer, and can appear more accurate than a model that answers all of them (Appendix H). The worst-case penalty prevents this: an invalid completion receives the maximum possible error over the admissible target range. This design follows the unanimous advice of the NeurIPS 2026 reviewers, who asked us to account for invalid completions with a worst-case penalty and to provide qualitative error analyses that distinguish prediction errors from failures to process the inputs or to follow the output format. Both are in the current version (Appendix H and Appendices I–K). We do not present WCP-RMSE as pure prediction error. Section 5 states that the visual and numerical results "reflect both predictive error and the ability to return a complete, machine-readable answer," and completion rate (CR) is reported next to every WCP-RMSE for this reason.

RMSE over valid outputs. Another reviewer made the same suggestion, and we are now computing RMSE over valid outputs only and will update the result tables soon, keeping CR and WCP-RMSE next to it.

The tool-free protocol is deliberate. Tool use, search, and code execution are disabled on purpose (Section 4.1), for two reasons. First, it keeps the comparison fair: the ten models are served through different provider stacks (Appendix O), and the tools available on each stack cannot be kept identical. Second, without Internet access a model cannot look up public sea-ice records at inference time, which minimises data leakage.


We hope these clarifications resolve the concerns about how the manuscript was prepared, and we respectfully ask the reviewer to reconsider the assessment in light of them. We are glad to answer any further questions.

Reviewer_JDw4OpenReview ↗

Please do not resubmit this manuscript in its current state to another venue and waste further community reviewing resources

First, I was genuinely angered upon receiving the authors' response. I fail to understand why the authors would waste time submitting a rebuttal for a manuscript that is so blatantly pieced together and AI-generated. I strongly urge the authors that, if they have any genuine respect for their own work and the data they collected, they must carefully write and vet the manuscript by hand rather than handing it over to AI to cobble together. Far from dispelling my concerns, the authors' response only further confirms my Strong Reject assessment. The specific issues are itemized below:

1、Q1 claims the authors reviewed all text and appendices before submission, yet App. N.2 (p. 35) literally states "The survey was added during the limited rebuttal period in response to a reviewer request".Splicing prior NeurIPS 2026 rebuttal text verbatim into an ICLR submission without basic editing proves that the manuscript was never properly vetted prior to submission.

2、Citations are padded indiscriminately or misattributed: citing RFC 4180 for CSV (p. 3), a 1990 paper for netCDF (p. 3), a 1992 military report for MGRS (p. 14), and the 1979 Belmont Report for a web-survey consent checkbox (p. 35). Citing Hoffman et al. (2023) for Attention-LSTM when that paper does not even use Attention-LSTM; citing a social-media keyphrase extraction paper (Devika et al., 2021) for Sentence-Transformer cosine similarity (p. 27); and listing Singh et al. (2025) with arXiv:2601.03267 (a January 2026 arXiv ID, p. 13). Meanwhile, basic typesetting is broken: missing macro spaces produce 13 concatenated strings such as SEAICE-BENCHcomprises, SEAICE-BENCHas, and SEAICE-BENCH53,785 starting in the Abstract; and §4 (p. 4, line 203) calls all evaluated models "frontier proprietary LLMs and MLLMs" one paragraph before stating three are open-weight.

3、On p. 10 (lines 513–518), the Generative AI Disclosure asserts: "We have not used generative AI tools for help develop theoretical models... clean and reformat dataset... interpret results. Formulate mathematical claims, provide critical ingredients for proving mathematical claims, are not applicable to this work." First, claiming not to have used GenAI to "clean and reformat dataset" is factually false: Appendix D, Figure 8 (pp. 22–23) explicitly documents a "QA Refine Prompt" instructing an LLM editor to strip source-document references, rewrite questions/answers/explanations for self-containment, and preserve the JSON dictionary schema.

4、[AI-assisted verification, manually verified by the reviewer] Under Eq. (8) (p. 28), in North motion vectors, Gemini 3 Flash (CR=0.2%, WCP-RMSE=164.1) fixes B >= 164.1, making Gemini 2.5 Flash (61.22 at CR=40%, min 127.1), Qwen (90.66 at CR=14.5%, min 151.7), and DeepSeek (89.81 at CR=18.3%, min 148.3) mathematically impossible; App. H also omits B and units for motion vectors. Even for extent, where Eq. (7) sets B_North=17, B_South=20, Qwen South (1.825 < min 2.449), Gemini 2.5 Flash North (4.552 < min 4.656), and Kimi (10.600/11.999 < min 10.684/12.116) violate Eq. (8).

5、[AI-assisted verification, manually verified by the reviewer] Train/test splits, extent sample counts, motion grid sizes, and ± SD definitions are all omitted (Table 5's ± SD is merely itemwise sqrt{p(1-p)} ). Table 3's full-scene CRs (76.47%, 97.96%) cannot even arise from integer counts over 500 locations (382.35, 489.80). Disabling search also fails to prevent pretraining memorization: historical targets (2018–2024, e.g., June 2019 in App. J) predate all ten models' 2025–2026 cutoffs (Table 9), and Table 4 omits a basic seasonal climatology baseline.

6、[AI-assisted verification, manually verified by the reviewer] Section 3.4 states "we convert imagery to PNG" without specifying radiometric scaling from 16-bit Sentinel-2 JPEG2000 files; in Appendix I (p. 29), Gemini 2.5 Flash explicitly reports that "bands 1–12 and TCI are largely black or very dark," which is visually confirmed by the nearly pitch-black panels in Figure 4 (p. 16). Worse, in the anonymous GitHub repository linked on p. 10, the OpenRouter evaluation script (img_eval_openR.py, targeting qwen3.6-flash, deepseek-v4-flash, and kimi-k2.6) labels .png bytes as data:image/jpeg;base64 and passes message_content as an unflattened nested sublist inside content ([{"type": "text", ...}, message_content]). This broken API payload directly matches why those exact three OpenRouter models claim in Appendix I that "No images were provided in your message" (Kimi K2.6 CR = 0.54%) or refuse visual analysis (also contradicting §4.1's claim of "native support" across all models)—meaning the benchmark is measuring the authors' pipeline bugs rather than MLLM capability.

While I believe the research topic itself holds potential, I still strongly urge the authors to settle down and conduct rigorous scientific research and manual academic writing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.