acceptodds
Under review as a conference paper at ICLR 2027

OmniEarthSeg: Unified Instruction-Driven Segmentation for Heterogeneous Remote Sensing Imagery

Abstract

Multimodal large language models (MLLMs) enable instruction-driven segmentation of remote sensing (RS) imagery, allowing users to specify targets through natural language. However, existing approaches often rely on modality-specific preprocessing or fixed input configurations, limiting their ability to accommodate variable-channel observations. Their mask decoders also depend primarily on high-level semantic tokens, which provide limited explicit spatial cues for target localization. To address these challenges, we curate OmniGlobal-1.9M, a large-scale dataset comprising 1.9 million instruction-segmentation samples from over 800K globally distributed RS images acquired by 17 sensors and platforms, spanning RGB, multispectral, and synthetic aperture radar modalities. It provides diverse annotations, including multi-granular segmentation masks spanning a broad range of categories and long-form textual supervision that supports interpretable reasoning and response generation. Building on this dataset, we propose OmniEarthSeg, an MLLM that integrates the OmniSensor Tokenizer for unifying variable-channel inputs, the Latent Modality-Aware Modulator for enhancing the model's awareness of modality-specific characteristics, and Prompt-Guided Visual Enhancement for localizing task-relevant visual evidence before segmentation decoding. Extensive experiments demonstrate that OmniEarthSeg achieves state-of-the-art performance on six benchmarks spanning different sensing modalities and generalizes effectively to unseen datasets under zero-shot evaluation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Remote Sensing, Instruction-Driven Segmentation, Multimodal large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NvStOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

OmniEarthSeg addresses instruction-driven remote sensing segmentation by introducing OmniGlobal-1.9M, a dataset of 1.9 million samples covering RGB, multispectral, and SAR imagery, and a unified MLLM framework. It combines variable-channel input encoding, modality-aware modulation, and prompt-guided visual enhancement to improve target localization. Experiments show state-of-the-art performance across six benchmarks and strong zero-shot generalization to unseen datasets.

Critical strengths
  1. OmniEarthSeg supports RGB, multispectral, and SAR imagery with a single model, accommodating variable-channel inputs and incorporating modality-aware and prompt-guided modeling for segmentation.
  2. OmniGlobal-1.9M provides a substantial data contribution, with approximately 1.9 million samples from over 800K images. Evaluation on six benchmarks and three unseen datasets, together with extensive ablations and instruction-tuned baselines retrained on the same dataset, provides broad empirical support.
Critical weaknesses
  1. On multispectral benchmarks, baseline methods only use RGB channels, while OmniEarthSeg uses all available channels. This shows the benefit of the overall method, but it is unclear how much improvement comes from the architecture itself rather than the additional input information.
Questions for authors
  1. Could you compare the methods using the same input channels? For example, could you give a strong baseline access to all channels or test OmniEarthSeg using only three channels?
  2. Could you explain how the evaluation subsets in Table 10 were selected and why some samples were excluded?
3 comments
AuthorsOpenReview ↗

Response to Q1 & Q2

Q1. Matched-Input Comparisons.

A1: We thank the reviewer for this constructive comment. To address this concern, we conducted two matched-input comparison experiments:

(1) We retrained OmniEarthSeg on OmniGlobal-1.9M using the same three-channel input protocol as the competing methods, retaining only RGB bands for multispectral imagery and replicating single-channel SAR imagery into three channels (see Section 5.2.1 of the main paper). We then evaluated the retrained model on the multispectral and SAR benchmarks reported in Table 2 of the main paper. The results are presented below:

MethodBigEarthNet-S2 mIoUBigEarthNet-S2 oIoUOpenEarthMap-SAR mIoUOpenEarthMap-SAR oIoU
LISA34.461.515.536.4
PixelLM33.760.913.029.0
GeoPixel40.268.121.140.5
EarthMind38.366.915.135.7
SegEarth-R134.754.521.739.2
UniGeoSeg22.130.317.929.2
SegEarth-R236.650.113.127.5
OmniEarthSeg (Ours)42.867.728.848.5

Under the same three-channel input setting, OmniEarthSeg achieves the highest mIoU on both benchmarks and the highest oIoU on OpenEarthMap-SAR, demonstrating that its performance advantage is not solely attributable to access to additional spectral information.

(2) We equipped UniGeoSeg [1] with a multichannel input adapter following EarthMind's [2] pseudo-RGB frame representation, enabling it to process all available spectral bands rather than only RGB channels. We then trained both the adapted UniGeoSeg and OmniEarthSeg on the same OmniGlobal-1.9M dataset and evaluated them on the multispectral and SAR benchmarks reported in Table 2 of the main paper. The results are presented below:

MethodBigEarthNet-S2 mIoUBigEarthNet-S2 oIoUOpenEarthMap-SAR mIoUOpenEarthMap-SAR oIoU
EarthMind36.867.216.837.6
UniGeoSeg + Adapter41.166.918.731.7
OmniEarthSeg (Ours)50.277.628.147.1

OmniEarthSeg consistently outperforms both EarthMind and UniGeoSeg + Adapter across all evaluation metrics. Together, these experiments demonstrate that OmniEarthSeg maintains its performance advantage under matched-input comparisons, whether restricted to three-channel inputs or compared against models equipped with multichannel processing capabilities. We will incorporate these results and the corresponding analysis into the revised manuscript.

[1] UniGeoSeg: Towards Unified Open-World Segmentation for Geospatial Scenes. CVPR, 2026.

[2] EarthMind: Leveraging Cross-Sensor Data for Advanced Earth Observation Interpretation with a Unified Multimodal LLM. arXiv preprint, 2025.

Q2. Evaluation Subset Filtering Criteria.

A2: We thank the reviewer for raising this question. To ensure the reliability of the evaluation subsets in Table 10, we applied two quality-control criteria:

(1) Mask Annotation Quality: Samples with erroneous or incomplete annotations, including masks that do not fully cover the corresponding target objects, were excluded.

(2) Valid Image Area: For datasets requiring image cropping, samples with a valid-area ratio below 0.8 after cropping were excluded to avoid evaluating images with insufficient valid content.

We have clarified these filtering criteria and the reasons for sample exclusion in Appendix A.3.2 of the revised manuscript.

Reviewer_NvStOpenReview ↗

Thank you for your rebuttal.

Thank you for the additional experiments and clarifications. The matched-input results largely address my concerns. I encourage the authors to incorporate these results and the corresponding details into the revised manuscript. I maintain my weak accept recommendation.

Thank you so much for your thoughtful assessment and encouraging feedback. We sincerely appreciate your positive evaluation and continued support!

Reviewer_zH64OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OmniEarthSeg, a unified instruction-driven segmentation framework for heterogeneous remote sensing imagery, along with OmniGlobal-1.9M, a large-scale dataset covering RGB, multispectral, and SAR modalities. The framework integrates three key components: the OmniSensor Tokenizer (OST) for variable-channel input adaptation, the Latent Modality-Aware Modulator (LMAM) for modality-aware representation learning, and Prompt-Guided Visual Enhancement (PGVE) for improving target localization. Experiments on six benchmarks and three unseen datasets demonstrate segmentation performance and zero-shot generalization.

Critical strengths
  1. The paper introduces OmniGlobal-1.9M, a large-scale dataset spanning RGB, multispectral, and SAR imagery, with diverse segmentation annotations and textual supervision. This dataset represents a valuable contribution to instruction-driven remote sensing segmentation.

  2. The paper presents a clear and well-justified motivation for addressing the challenges of heterogeneous remote sensing imagery with varying channel configurations. The proposed unified framework provides a practical approach to instruction-driven segmentation across diverse sensing modalities.

Critical weaknesses

The fairness of the comparisons with existing OVSS methods and the rationale behind model selection during dataset construction could be further clarified. In addition, more comprehensive experimental evaluations and analyses are needed to better assess the effectiveness of the proposed method on larger-scale datasets.

Questions for authors

1.Fairness of comparisons with OVSS methods. In Table 2, OmniEarthSeg is compared with open-vocabulary semantic segmentation (OVSS) methods on six benchmarks: OpenEarthMap, iSAID, FLAIR, BigEarthNet-S2, OpenEarthMap-SAR, and HRSID. However, as shown in Table 6, these datasets are also used to construct OmniGlobal-1.9M, on which OmniEarthSeg is trained. In contrast, the OVSS baselines are evaluated using their original pretrained checkpoints without being trained on the evaluation datasets.This discrepancy in training data and supervision raises concerns about the fairness of the comparisons. The authors should further justify the comparison protocol.

2.Model selection criteria in dataset construction. In Figure 6, the authors employ several large models, including InternVL3-38B, GPT-4o, Qwen2.5-72B, and Intern-S1-Pro, to construct OmniGlobal-1.9M. However, the rationale for selecting these particular models remains unclear. The authors are encouraged to clarify the selection criteria and explain the respective roles of these models in the dataset construction process.

3.Evaluation on larger-scale datasets. In Table 2, several evaluation datasets, including OpenEarthMap, OpenEarthMap-SAR, and HRSID, are relatively small in scale, which may limit the ability of the current experiments to fully demonstrate the strong performance of OmniEarthSeg. The authors are encouraged to conduct additional experiments on larger-scale benchmarks to provide more convincing evidence of the proposed method's effectiveness.

4.OmniEarthSeg demonstrates the ability to process RGB, multispectral, and SAR imagery within a unified framework. However, it remains unclear whether the proposed method can simultaneously process and fuse inputs from multiple sensing modalities. The authors should clarify whether such joint processing is supported and, if so, provide additional experiments on multi-modal joint segmentation to further validate the proposed framework.

5.Effectiveness and computational overhead of LMAM. As shown in Table 4, incorporating LMAM into the combination of OST and PGVE yields only a marginal improvement in average mIoU (from 42.9% to 43.1%). This raises questions about whether the additional computational overhead introduced by LMAM is justified by its limited performance gain. The authors should provide a more detailed analysis of the trade-off between performance and computational efficiency and discuss whether LMAM could be removed to simplify the framework.

6.Typo. The x-axis label in Figure 1(c) should be corrected to “Number of Categories per Sample.”

4 comments
AuthorsOpenReview ↗

Response to Q1 & Q2

Q1. Fairness of OVSS Comparisons.

A1: We thank the reviewer for raising this important concern. We would like to clarify the following points:

(1) The training and evaluation data used in the experiments reported in Table 2 are strictly separated, with no overlap between the two sets, thereby preventing potential data leakage during evaluation. The dataset splitting details are provided in Table 10.

(2) The zero-shot experiments reported in Table 3 already provide evidence of the generalization capabilities of both the OVSS baselines and our model. Specifically, these evaluations are conducted on data sources that are not included in the original training data. The results show that our model continues to outperform the OVSS baselines under this zero-shot setting, demonstrating its strong generalization ability.

(3) To further investigate this issue and provide a more controlled comparison, we selected SkySense-O as a representative OVSS baseline and extensively fine-tuned it on OmniGlobal-1.9M. We then evaluated both methods across three benchmarks covering different data modalities: RGB (ISAID), multispectral (FLAIR), and SAR (OpenEarthMap-SAR).

The results are reported below:

MethodISAID mIoUISAID oIoUFLAIR mIoUFLAIR oIoUOpenEarthMap-SAR mIoUOpenEarthMap-SAR oIoU
SkySense-O45.161.558.068.523.345.0
OmniEarthSeg (Ours)66.294.758.669.028.147.1

Despite extensive fine-tuning on OmniGlobal-1.9M, SkySense-O consistently underperforms OmniEarthSeg across all three benchmarks and both evaluation metrics. These results suggest that OmniEarthSeg's performance advantage cannot be explained solely by differences in training data exposure. We will incorporate this additional comparison and the corresponding analysis into the revised manuscript.

Q2. Model Selection in Dataset Construction.

A2: We thank the reviewer for raising this concern. The model selection in our dataset construction pipeline is primarily guided by the specific requirements of each stage, considering task suitability, generation quality, and computational efficiency. We clarify the respective roles and selection rationales as follows.

  • Mask Filtering. Following UniGeoSeg [1], we employ InternVL3-38B as a model-based evaluator to assess candidate annotations in terms of target validity, category consistency, and spatial alignment, leveraging an established filtering strategy for remote sensing segmentation annotations.

  • Instruction Construction. We adopt GPT-4o with task-specific prompt templates to generate four types of instructions, including scene-analysis segmentation, direct semantic segmentation, attribute-analysis segmentation, and category-free scene description. This choice also follows the established practice in UniGeoSeg[1].

  • CoT and Answer Generation. Given an image, its validated instruction, and the target categories derived from the corresponding segmentation mask, we employ InternVL3-78B to generate the chain-of-thought (CoT) rationale and final answer. This stage involves substantial visual-token processing and long-form text generation, with the combined CoT rationale and answer typically reaching approximately 350 words. We select InternVL3-78B to balance long-form multimodal generation quality and computational cost, avoiding the substantial expense of proprietary models at this scale.

  • Annotation Quality Assessment. To further ensure annotation reliability and mitigate model-specific hallucinations, we adopt a two-stage quality auditing strategy. In the first stage, we employ Qwen2.5-VL-72B, which belongs to a different model family from InternVL3, to independently evaluate the CoT rationale and final answer. Using a different model family provides an additional assessment perspective and helps mitigate potential model-specific biases. In the second stage, we employ Intern-S1-Pro to conduct a final review of the samples that pass the first-stage evaluation. This two-stage strategy helps maximize data quality while keeping the overall auditing costs under control.

We have clarified the model selection criteria and the respective roles of these models in the Appendix A.1.3 of the revised manuscript.

[1] UniGeoSeg: Towards Unified Open-World Segmentation for Geospatial Scenes. CVPR, 2026.

AuthorsOpenReview ↗

Response to Q3-Q6

Q3. Evaluation on Larger-Scale Datasets.

A3: We thank the reviewer for this suggestion. To further validate the effectiveness of OmniEarthSeg on larger-scale benchmarks, we conducted additional evaluations on two datasets:

  • Chesapeake (Multispectral): 25,779 test image–text pairs with 61,881 target masks.

  • Globe230K (RGB): 22,237 test image–text pairs with 40,856 target masks.

The results are summarized below.

MethodChesapeake mIoUChesapeake oIoUGlobe230K mIoUGlobe230K oIoU
SegEarth-R248.051.952.055.8
UniGeoSeg32.438.243.946.3
OmniEarthSeg (Ours)64.887.161.469.8

OmniEarthSeg consistently outperforms both competing methods across the two datasets and all evaluation metrics. In particular, it improves upon the strongest baseline by 16.8 and 9.4 mIoU points on Chesapeake and Globe230K, respectively. These results provide additional evidence of its effectiveness on larger-scale segmentation benchmarks.

Q4. Multimodal Joint Segmentation.

A4: We thank the reviewer for raising this interesting question. OmniEarthSeg supports variable-channel inputs of up to 13 channels, allowing spatially aligned observations from different sensing modalities to be jointly processed.

Specifically, to investigate the impact of different fusion stages on multimodal segmentation, we designed the following two fusion strategies:

(1) Image-level Fusion. At the input stage, spatially aligned RGB and SAR images are concatenated along the channel dimension to construct a unified four-channel RGB-SAR image, which is subsequently fed into the visual encoder for feature extraction.

$ \operatorname{Concat}_{C}(\mathrm{R},\mathrm{G},\mathrm{B},\mathrm{SAR}) \in \mathbb{R}^{H\times W\times 4} $

(2) Feature-level Fusion. RGB and SAR images are separately processed by the shared OmniSensor Tokenizer (OST) to obtain their respective patch-level representations. The features from the two modalities are then combined through element-wise addition to achieve cross-modal feature fusion. The fused representation is then processed by the subsequent visual encoder blocks. Formally,

ERGB=OST(IRGB)E_{RGB} = OST(I_{RGB})

ESAR=OST(ISAR)E_{SAR} = OST(I_{SAR})

$F_{final} = Encoder(E_{RGB} + E_{SAR} ) $

Here, the patch-level features from the two modalities share the same spatial positions and feature dimensions, allowing element-wise addition.

We compare the two fusion strategies against a baseline that processes RGB and SAR images separately. All experiments are conducted on the OpenEarthMap-SAR dataset, which provides spatially aligned RGB-SAR image pairs. All variants are trained and evaluated under the same data splits.

MethodmIoUoIoU
RGB57.975.0
SAR28.147.1
Image-level Fusion69.985.6
Feature-level Fusion70.585.6

Both fusion strategies outperform the RGB-only and SAR-only baselines, with feature-level fusion achieving the highest mIoU of 70.5%, a 12.6-point improvement over RGB-only segmentation. These results demonstrate that OmniEarthSeg can jointly process and effectively leverage complementary information from aligned RGB and SAR observations.

We will incorporate these additional results into the revised manuscript.

Q5. LMAM Effectiveness and Efficiency.

A5: We thank the reviewer for raising this concern. To evaluate the trade-off between segmentation accuracy and computational efficiency, we measured the inference time with and without LMAM under identical evaluation settings on the Chesapeake dataset.

As shown below, introducing LMAM increases the inference time from 3.47 to 3.52 seconds per image, corresponding to an additional overhead of only 0.05 s/image (approximately 1.4%).

Metricw/o LMAMw/ LMAMΔ
Inference Time3.47 s/img3.52 s/img+0.05 s/img

Although LMAM provides a modest improvement of 0.2 percentage points in average mIoU (Table 4), it introduces minimal additional inference overhead. Therefore, LMAM can be removed to simplify the framework, but we retain it in the full model to benefit from its additional accuracy improvement at a relatively low computational cost.

We will incorporate these efficiency measurements and the corresponding trade-off analysis into the revised manuscript.

Q6. Figure Label Correction.

A6: We thank the reviewer for pointing out this labeling error. We have corrected the x-axis label of Fig. 2(c) to “Number of Categories per Sample” in the revised manuscript.

Thank you for the authors' response. My concerns have been addressed, and I will keep my rating of weak acceptance.

Thank you so much for acknowledging our clarifications. We truly appreciate your favorable assessment and encouraging response!

Reviewer_NNgcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OmniGlobal-1.9M and OmniEarthSeg for instruction-driven segmentation of heterogeneous remote sensing imagery. The dataset contains 802,812 images and 1,935,675 instruction-response samples across RGB, multispectral, and SAR modalities. OmniEarthSeg introduces three components for remote sensing segmentation. The OmniSensor Tokenizer converts variable-channel inputs into unified visual tokens. The Latent Modality-Aware Modulator injects visual modality cues into language representations. Prompt-Guided Visual Enhancement uses instruction semantics to refine visual features before mask decoding. A dual-branch CNN–ViT encoder combines local spatial details and global context through multi-scale feature fusion. Experiments cover six segmentation benchmarks and three additional datasets for zero-shot evaluation.

Critical strengths
  1. OmniGlobal-1.9M provides a large-scale resource for instruction-driven remote sensing segmentation. It combines RGB, multispectral, and SAR imagery with segmentation masks and textual supervision.
  2. OST allows the model to use additional spectral bands without increasing the visual token sequence length. It embeds each channel separately and aggregates the channel tokens into one token per spatial patch, as defined in Eqs. 1–2.
  3. Experimental results show consistent improvements over the compared methods. Table 2 reports 28.1 mIoU on OpenEarthMap-SAR, compared with 21.7 for the best baseline. Under zero-shot evaluation, Table 3 reports 46.7 versus 44.1 on DLRSD and 43.1 versus 37.7 on AIR-PolSAR.
  4. The ablation studies support the effectiveness of the proposed modules. Table 4 shows gains from LMAM and PGVE, and Table 5 reports improvements with two MLLM backbones. Appendix A.4 further examines the effects of training data scale and visual encoder design.
Critical weaknesses
  1. The novelty of OST and PGVE is not sufficiently clarified. OST embeds individual channels and uses a learned query to aggregate channel tokens at each spatial position. ClimaX [1] already introduced this tokenization and aggregation design. Panopticon [2] applies the same aggregation mechanism to heterogeneous remote sensing imagery. PGVE uses visual queries, textual keys and values, and gated residual fusion. Grounding DINO [3] already uses these operations in its feature enhancer. LAVT [4] also combines pixel-word attention with gated visual fusion for segmentation. It would be better to distinguish the proposed method from these previous works.

  2. The EarthMind comparison does not use its supported multispectral input. Lines 365–370 state that the comparison methods receive only RGB channels from multispectral images. However, EarthMind (Shu et al., 2025) supports multispectral inputs. In Table 2, OmniEarthSeg uses multispectral inputs, while EarthMind uses only RGB channels. This difference makes the comparison unfair. It would be better to explain why EarthMind’s supported multispectral input was not used.

  3. The explanation contradicts the segmentation results. In the middle example of Fig. 28, the answer states, “No clear signs of Farmland are visible in this scene,” while both displayed segmentation maps contain regions labeled Farmland. This implies the model’s explanation is unreliable.

  4. There are several presentation problems:

    • Lines 375–376 describe this baseline as “fine-tuned on remote sensing data.” However, SegEarth-OV3 (Li et al., 2025c) is a training-free method.
    • Table 2 reports 58.6 mIoU on FLAIR, while Table 11 reports 59.0.
    • Fig. 2(c) is titled “Categories per Sample,” but the horizontal axis is “Number of Instances per Sample.”
    • The bar heights in Fig. 14 appear incorrect. They sum to approximately 119%, 125%, and 122% for prompts, CoT rationales, and answers, respectively, exceeding 100%.

[1] Nguyen et al., ClimaX: A Foundation Model for Weather and Climate. ICML, 2023.

[2] Waldmann et al., Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation. CVPR Workshops, 2025.

[3] Liu et al., Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. ECCV, 2024.

[4] Yang et al., LAVT: Language-Aware Vision Transformer for Referring Image Segmentation. CVPR, 2022.

Questions for authors

Please refer to the critical weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to Q1

Q1. Novelty of OST and PGVE.

We thank the reviewer for this insightful comment. We acknowledge that OST and PGVE build upon established attention and feature fusion mechanisms. Their distinctions from prior works lie in how these mechanisms are designed and integrated to address heterogeneous remote sensing inputs and instruction-driven pixel-level segmentation.

(1) OST vs. ClimaX and Panopticon: The key distinction between OST and ClimaX lies in the mapping between input channels and embedding kernels. ClimaX associates embedding kernels with predefined physical variables, requiring additional embedding parameters for previously unseen variables. In contrast, OST employs 13 channel-position-specific convolution kernels and a channel-indexing strategy determined by the number and order of input channels. Specifically, 13-channel inputs use kernels 1–13, whereas for inputs with fewer than 13 channels, the cc-th channel uses the kernel at position c+1c+1. This allows inputs with varying channel counts to share a fixed set of learned embedding parameters without introducing new kernels or requiring additional fine-tuning.

Compared with Panopticon, which shares a patch projection across channels and incorporates explicit wavelength or SAR attribute encodings, OST learns channel-position-specific projections without requiring such metadata. These features are subsequently aggregated through channel-adaptive attention into spatially aligned visual tokens. Thus, OST provides a unified input representation for RGB, multispectral, and SAR imagery through a fixed parameterization and adaptive channel aggregation.

(2) PGVE vs. Grounding DINO and LAVT: Although PGVE shares cross-modal attention and gated fusion operations with prior works, its integration into the segmentation framework differs in several important aspects. Grounding DINO employs bidirectional image-text feature enhancement during encoding to support object detection. In contrast, PGVE operates after visual encoding and before mask decoding, using multi-scale visual features as queries and LMAM-calibrated instruction representations as keys and values. The resulting features are modulated by a learnable channel-wise gate initialized to 0.01 and jointly decoded with the [SEG] token for pixel-level mask prediction.

Unlike LAVT, which progressively integrates BERT word features into intermediate visual representations using spatially and channel-wise varying gates, PGVE enhances the aggregated multi-scale visual features using instruction representations from the language model. This design directly connects instruction-conditioned visual enhancement with mask decoding while retaining multi-scale spatial information.

Overall, while the underlying attention and fusion operations are established, OST and PGVE differ from these methods in their channel parameterization, feature interaction stages, and integration with instruction-driven segmentation. Together, they enable OmniEarthSeg to process heterogeneous remote sensing imagery and generate instruction-conditioned segmentation masks within a unified framework.

Manuscript Revision. To better clarify these distinctions, we have added explicit comparisons with ClimaX and Panopticon in Section 4.2, and with Grounding DINO and LAVT in Section 4.4 of the revised manuscript. These additions clarify the specific design choices underlying OST and PGVE and their roles in unified instruction-driven segmentation across heterogeneous remote sensing imagery.

AuthorsOpenReview ↗

Response to Q2 & Q3

Q2. Fair Comparison with EarthMind.

A2: We thank the reviewer for pointing out this issue. EarthMind processes multispectral imagery by grouping every three consecutive bands into a pseudo-RGB frame, treating the resulting frames as a video sequence. If the final group contains fewer than three bands, the last available band is replicated to complete the frame. In our original experiments, we adopted a consistent three-channel input protocol for the competing methods, extracting RGB channels from multispectral imagery and replicating single-channel SAR imagery into three channels. However, we acknowledge that this protocol does not fully exploit EarthMind's multispectral processing capability and may underestimate its performance.

To address this concern, we have re-evaluated EarthMind using its officially supported multispectral input format, retaining all spectral bands through the pseudo-RGB frame representation. Specifically, we fine-tuned EarthMind on OmniGlobal-1.9M using this input format and evaluated it on two multispectral benchmarks. The results are presented below.

MethodBigEarthNet-S2 mIoUBigEarthNet-S2 oIoUFLAIR mIoUFLAIR oIoU
EarthMind (RGB input)38.366.949.060.6
EarthMind (officially supported multispectral input format)36.867.250.763.4
OmniEarthSeg (Ours)50.277.658.669.0

OmniEarthSeg consistently outperforms EarthMind under its supported multispectral input setting, achieving improvements of 13.4 and 7.9 mIoU points on BigEarthNet-S2 and FLAIR, respectively. These results demonstrate that the performance advantage of OmniEarthSeg persists when EarthMind is allowed to exploit the full multispectral observations.

We will incorporate these additional results into the final manuscript to provide a more comprehensive and fair comparison with EarthMind.

Q3. Consistency of Qualitative Results.

A3: We thank the reviewer for identifying this inconsistency. We acknowledge that the generated explanation in Fig. 28 does not fully align with the displayed segmentation results. Although the segmentation predictions are accurate, achieving an mIoU of 97.23% and an oIoU of 97.49%, the textual explanation incorrectly describes the presence of farmland. Since Fig. 28 is intended as a qualitative illustration in the Appendix, we have replaced this example with another representative case in which the generated explanation and segmentation results are consistent. We will also carefully review the remaining qualitative examples to avoid similar inconsistencies.

AuthorsOpenReview ↗

Response to Q4

Q4. Corrections to Presentation Issues.

A4-(1): We thank the reviewer for pointing out this inaccurate description. We acknowledge that SegEarth-OV3 is a training-free method rather than a model fine-tuned on remote sensing data. We have corrected the corresponding statement in Section 5.2.1 of the revised manuscript as follows:

"Notably, despite being built upon SAM3 (Carion et al., 2026), which offers advanced visual representation and segmentation capabilities, SegEarth-OV3 (Li et al., 2025c) is still outperformed by OmniEarthSeg."

A4-(2): We thank the reviewer for identifying this inconsistency. Upon rechecking the original experimental records, we found that the gIoU value (59.0401%) was mistakenly reported as mIoU in Table 11. The correct mIoU is 58.5530%, consistent with the 58.6% reported in Table 2.

We have corrected the corresponding values in Table 11 and updated the average mIoU for the 100% training-data setting from 66.2% to 66.1% in both the table and the accompanying discussion. These corrections have been incorporated into the revised manuscript.

A4-(3): We thank the reviewer for identifying this labeling inconsistency. Fig. 2(c) presents the distribution of the number of categories per sample. We have corrected its horizontal axis label to “Number of Categories per Sample” in the revised manuscript.

A4-(4): We thank the reviewer for pointing out this issue. The underlying frequency statistics were calculated correctly, but the bar heights in Fig. 14 were nonlinearly scaled to improve the visibility of smaller values, resulting in a misleading visual representation of the percentages. We acknowledge that this visualization was inappropriate and have corrected Fig. 14 to display the actual percentages without nonlinear scaling in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.