acceptodds
Under review as a conference paper at ICLR 2027

Web2Earth: Extending Web Vision Foundation Models Across Remote Sensing Modalities

Abstract

Vision foundation models pretrained on large-scale web imagery learn rich visual representations, yet their RGB-centric training leaves a gap in their ability to handle other sensing modalities. Motivated by the pursuit of remote sensing foundation models that perform well across modalities and tasks, we ask whether this existing knowledge can be extended beyond RGB while also improving optical performance. We introduce Web2Earth, a framework that builds on a web-pretrained encoder to pursue broad remote sensing utility through joint RGB self-distillation and cross-modal knowledge transfer. We design complementary patch and relational distillation objectives to transfer local spatial knowledge and scene-level relationships from RGB to other sensing modalities. Adaptation uses 32k RGB images and approximately 5k co-registered image pairs, while updating only approximately 2% of the model parameters. Evaluations across diverse optical and SAR tasks demonstrate strong cross-modal transfer, with leading results on the main optical evaluations and the highest average score across six SAR evaluations among the compared models. Notably, joint RGB adaptation and cross-modal alignment improve optical performance over the starting web-pretrained model while extending its utility to SAR. These results support extending the knowledge in large-scale web-pretrained models toward remote sensing foundation models with strong performance across both modalities and tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Remote Sensing Foundation Models, Multimodal Representation Learning, Parameter-Efficient Adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KjXJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents Web2Earth, a framework for extending a web-pretrained vision encoder to remote sensing imagery. The approach jointly optimizes RGB self-distillation and two cross-modal losses. CMRD encourages SAR representations to reproduce the RGB teacher’s inter-scene similarity structure, while CoPaD matches teacher and student prototype predictions at corresponding patch locations. Initialized from DINOv3, the resulting model is assessed through frozen-encoder evaluations covering classification, segmentation, change detection, and scene retrieval.

Critical strengths

The study explores a useful alternative to large-scale remote sensing pretraining by adapting an existing web-trained model with relatively little additional data and a small number of backbone updates. Retaining optical performance while improving SAR representations is practically valuable, as is the ability to process either modality independently during inference.

The experimental coverage is broad, spanning multiple tasks, datasets, and SAR sensors not encountered during adaptation. Evaluations without task-specific encoder training help assess the transferability of the learned features. The same-data continual-pretraining baseline and separate evaluations of the two cross-modal objectives provide useful evidence that paired supervision contributes beyond additional training alone.

Critical weaknesses

The technical contribution appears incremental when viewed in the context of existing cross-modal representation learning. Learning non-RGB representations from paired RGB observations was established by Cross Modal Distillation for Supervision Transfer (CVPR 2016). Relational KD and similarity-preserving KD already motivate transferring relationships rather than absolute feature coordinates, and DistilVPR (AAAI 2024) applies relational supervision across image and point-cloud representations. Correspondence-based local transfer is also well explored in SLidR (CVPR 2022) and foundation-model adaptation methods such as EventSAM (CVPR 2024). CMRD largely follows this established similarity-matching paradigm, while CoPaD uses a conventional prototype-based distillation objective at spatially aligned locations. Although their integration is sensible, the paper does not adequately explain what methodological advance it introduces beyond adapting these familiar ingredients to optical–SAR data. The contribution therefore reads primarily as an empirical adaptation study.

Several central interpretations are not fully supported by controlled comparisons. In Table 5, the raw feature baseline excludes projection heads, whereas CoPaD introduces both projection heads and prototype-distribution matching. This comparison cannot determine whether the improvement comes from the proposed prediction-space objective or simply from allowing a learned projection. Moreover, matching within-modality similarity matrices does not by itself align the modalities for direct retrieval: CMRD permits orthogonal transformations of the student representations, but evaluation relies on cross-modal cosine similarities between CLS tokens. Component-level retrieval experiments are needed to explain how this ambiguity is resolved.

Questions for authors

Beyond their application to optical–SAR imagery, what substantive advances do CMRD and CoPaD offer over existing relational and spatial correspondence-based distillation methods? What are the individual contributions of CMRD and CoPaD to RGB-to-SAR retrieval?

2 comments
Reviewer_KjXJOpenReview ↗

RE: Rebuttal

Thank you for the detailed responses and additional experiments. The projected-feature alignment control and component-level retrieval ablations substantially address my concerns about the experimental comparisons. I also appreciate the acknowledgment that CMRD does not guarantee cross-modal coordinate alignment. These additions strengthen the empirical evidence.

Thank you very much for your thoughtful follow-up and for carefully considering our additional experiments. Your comments helped us strengthen the experimental comparisons and clarify both the contribution and limitations of our approach. We sincerely appreciate the time and care you have devoted to reviewing our work. We will incorporate these additional results and clarifications into the revised manuscript. We hope the strengthened evidence helps convey the value of our central contribution: a simple, lightweight framework that extends a web-pretrained encoder to SAR while preserving strong optical transfer. Thank you again for the constructive discussion and for helping us improve the paper.

Reviewer_omLrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Web2Earth, a lightweight framework for adapting web-pretrained vision foundation models to remote sensing. It uses RGB self-distillation and two cross-modal objectives, CMRD and CoPaD, to transfer knowledge from RGB to SAR. The method updates only about 2% of the DINOv3 backbone parameters. Experiments on optical and SAR tasks show improved performance.

Critical strengths

S1. Reasonable cross-modal supervision design. CMRD transfers scene-level relationships, while CoPaD transfers patch-level predictions. This design avoids directly aligning RGB and SAR feature vectors. The ablation results support the effectiveness of these objectives.

S2. Broad experimental evaluation. The paper evaluates multiple optical and SAR tasks. It also includes unseen SAR sensors and cross-modal retrieval. The performance improvements are generally consistent, especially for RGB-to-SAR retrieval.

Critical weaknesses

W1. Limited methodological novelty. CMRD and CoPaD mainly extend existing relational and patch-level distillation methods to RGB-SAR data. The overall framework also follows the standard DINOv3 self-distillation paradigm. The paper lacks comparisons with standard paired cross-modal contrastive learning and other distillation methods. Therefore, the unique benefits of the proposed objectives are not fully established.

W2. Incomplete analysis of the performance gains. The paper reports improvements in both RGB and SAR performance. However, it lacks an RGB-only adaptation baseline. It is unclear how much of the optical improvement comes from RGB domain adaptation rather than cross-modal learning. Moreover, the large RGB-to-SAR retrieval gains are not supported by separate CMRD and CoPaD ablations. The contributions of these objectives to cross-modal alignment remain unclear.

Questions for authors

Q1. Could the authors add an RGB-only self-distillation baseline? How much of the optical improvement comes from cross-modal learning rather than RGB domain adaptation?

Q2. What are the main differences between CMRD/CoPaD and existing relational or patch-level distillation methods? Could the authors compare them with standard RGB-SAR contrastive learning under the same training budget?

Q3. Could the authors report separate CMRD and CoPaD ablations for RGB-to-SAR retrieval? How does CMRD improve cross-modal retrieval when it only matches within-modality scene similarities?

Q4. How sensitive is CoPaD to imperfect RGB-SAR registration? Could the authors also evaluate the effect of RGB-based SAR statistics matching during training?

I'm willing to raise my score to 3 points after you provide a convincing answer to my question.

Reviewer_9dx1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a framework for continual pretraining of Dinov3 with RGB/SAR satellite images. In particular, the authors present a framework to leverage paired data from the two modalities by introducing cross-modal alignment loss terms between student and teacher. These are cross-modal extensions of the DINO and iBot loss. In addition, they also add modality-specifc self-supervision loss terms. The encoder is initialized by Dinov3 and further fine-tuned with LORA. The results look strong across RGB and SAR downstream datasets, outperforming existing models and the Dinov3 baseline.

Critical strengths
  • Extensive Evaluation on optical and SAR datasets, showing strong results across the datasets.
  • Cheap and simple student-teacher framework to improve upon Dinov3 for satellite imaging tasks.
Critical weaknesses
  • Model is currently limited to RGB-SAR pairs.
  • The novelty of the framework is limited. The main contribution is a cross-modal DINO prototype matching, which is not novel [1].

[1]: Wang, Yi, Conrad M. Albrecht, and Xiao Xiang Zhu. "Self-supervised vision transformers for joint SAR-optical representation learning." IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium. IEEE, 2022.

Questions for authors
  • The authors always mention non-RGB modality. Does the approach generalize beyond SAR?
  • To me, it is surprising that Dinov3-SAT is outperformed by Dinov3 and Dinov2 on remote sensing tasks? Do you have any explanation for that?
  • What is the number of prototypes that you use? Did you use a warm-up for the training as the encoder are initialized with Dino backbone and the heads are randomly initialized?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.