acceptodds
Under review as a conference paper at ICLR 2027

SAREO-FM: Decoupled Semantic Supervision for SAR-EO Foundation Models

Abstract

Synthetic aperture radar (SAR) and electro-optical (EO) imagery provide complementary observations: SAR enables day-and-night, weather-resilient sensing, whereas EO provides rich appearance and fine-grained semantic cues. We introduce SAREO-FM, which avoids forcing a single token stream to serve two distinct roles: modality tokens preserve how each sensor observes the scene through masked reconstruction, while learnable semantic queries capture what the scene contains under guidance from a pretrained vision foundation model (VFM). By jointly encoding these queries with SAR and EO tokens, the queries acquire modality-grounded semantic context, while the modality-token outputs remain the explicit targets of masked reconstruction. This design assigns semantic and reconstruction supervision to separate token streams while preserving their interaction within the shared encoder. Pretrained on the million-scale SAR-1M corpus, SAREO-FM achieves state-of-the-art unimodal transfer for both SAR-only and EO-only inputs, while delivering substantial gains from joint SAR-EO observations on tasks that benefit from complementary sensing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Remote Sensing, SAR–EO Representation Learning, Multimodal Foundation Models, Masked Autoencoders, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SfKwOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a unified SAR–EO foundation model that decouples high-level semantic alignment from modality-specific reconstruction within a single masked autoencoding framework. The authors introduce learnable, spatially organized semantic queries that are aligned patch-wise to features from a frozen EO vision FM (DINOv3), while SAR and EO image tokens are used exclusively for masked pixel reconstruction. All three streams interact in a shared encoder.

Critical strengths

The authors address a practically important challenge in geospatial foundation modeling: learning modality-specific details and cross-modal semantics without conflict

  • As technical novelty, the authors propose a decoupled semantic supervision via persistent learnable queries. This can be considered as a relevant answer to the “one stream, two duties” issue common in MAE-style multimodal pretraining. In addition, the unified three-stream encoder (SAR tokens, EO tokens, and semantic queries) together with modality-specific decoders and query-only alignment is a principled architectural separation that preserves cross-stream interaction while preventing supervision entanglement. This seems novel to me. The mixed modality masking strategy is well designed to expose the same encoder to SAR-only, EO-only, and paired inputs. This may improve the robustness to missing modalities.

  • Experiments include fair controlled comparison against SARMAE with matched backbones, datasets, and protocols isolating the effect of raw EO inclusion and the decoupled objective. Evaluations span SAR-only, EO-only , and joint SAR–EO , plus a qualitative dense-prediction example on BRIGHT. The ablation study is fair enough to show the individual contribution of each component (MAE, DINOv3 supervision, decoupled supervision, raw EO modality, etc.)

  • The motivation is well-articulated, and the method is presented with clear mathematical detail, enabling reproducibility.

Critical weaknesses

There are some technical limitations or concerns:

  • From a methodological point of view, there is a big dependence on a large frozen EO teacher (DINOv3) and on paired SAR–EO data for alignment. This may raise questions about scalability to domains with limited or misregistered EO partners, and about potential optical bias in the learned semantics. Also the semantic queries are fixed in count and unmasked. The authors did not explore sensitivity to query number, masking, or alternative alignment destinations (e.g., a small set of global queries vs. full patch grid).

  • From an experimental point of view, quantitative dense prediction results are quite limited. Only qualitative BRIGHT examples are shown, leaving open how well the approach transfers to pixel-level tasks where SAR excels. Comparisons to broader multimodal GFMs (e.g., OmniSat, RAMEN, SpectralEarth-FM) are not reported, even if not strictly SAR-focused, which would better contextualize the contribution within current EO multimodal pretraining trends. EO transfer is evaluated mainly on EuroSAT. Additional EO benchmarks (e.g., RESISC45, AID in the main results) are mentioned but not shown in tables, limiting the EO-branch evidence...

  • For clarity, some training-mode details (e.g., exact share of unpaired samples, handling of imperfect coregistration, and cloud/noise effects in teacher) could be made more explicit in the paper. The downstream usage of semantic-query features (pooling, readout strategy) could be specified more clearly for reproducibility.

  • For positioning in the SOA, the paper would benefit from deeper discussion and experimental contrast to any-to-any and resolution-aware multimodal MAE/JEPA paradigms (e.g., MultiMAE/4M, OmniSat, RAMEN) that also address missing modalities and decoders per modality.

Questions for authors

How robust is the approach to SAR–EO misregistration and cloud contamination in teacher features? Can you report performance under synthetic misalignment or cloud‑mask filtering of teacher inputs? How does the method perform on quantitative dense prediction tasks (e.g., Sen1Floods11, PASTIS, BRIGHT) with SAR-only, EO-only, and joint inputs?

5 comments
AuthorsOpenReview ↗

Response to Reviewer SfKw (1/5)

We sincerely thank the reviewer for the careful reading and thoughtful, constructive feedback. Your suggestions have helped us strengthen our evaluations of dense prediction and EO transfer, clarify the training and readout protocols, and better explain the robustness limitations and relation to prior multimodal work.

W1. Teacher dependence, modality masking, and query design

(i) Teacher dependence. As reported in our response to Reviewer qBhv, W2, DINOv3-7B/L/B yield average OA of 71.47/71.32/71.18, a range of only 0.32 points. This is smaller than the 2.90-point gain across the query/supervision variants in qBhv W1/Q1, suggesting that our gains are not primarily explained by teacher scale.

(ii) Modality masking. The ablation in our response to Reviewer 7NcP, W3 gives average OA of 71.50 for the full configuration, 40.91 without modality dropping (−30.59 points), and 70.24 without shared spatial masking (−1.26 points). These results support explicitly training for varying sensor availability, with modality dropping making the largest contribution.

(iii) Alignment destination and query granularity. The SAR-only controls in Reviewer qBhv, W1/Q1 show successive average OA gains of +0.81 points from adding queries, +1.34 from redirecting alignment to queries at the same 64 positions, and +1.07 from extending alignment to all 256 positions. All variants average the alignment loss over their supervised spatial positions. These controls support both the supervision destination and spatial coverage.

We additionally evaluate 4×4 (16-query) and 1×1 (global-query) pooled configurations in the SAR+EO ablation, compared with the full 16×16 query grid. Scores are OA (%); Avg. is the unweighted mean across benchmarks, and Δ is the change in Avg. from the preceding row, in percentage points.

Query configurationMSTARFUSAR-ShipSAR-ACDAvg.Δ
Full 16×16 grid: 256 queries (reference)78.4184.0052.0971.500.00
Pooled 4×4 grid: 16 queries64.8374.6942.5060.67−10.83
Pooled 1×1 grid: one global query60.0572.7941.5758.14−2.54

Pooling to 4×4 and 1×1 lowers average OA by 10.83 and 13.36 points relative to the full grid. This suggests that mixing features from different regions weakens the local information needed for semantic alignment. All queries remain unmasked.

AuthorsOpenReview ↗

Response to Reviewer SfKw (2/5)

W2. Quantitative dense prediction and additional EO benchmarks

We added quantitative dense prediction on BRIGHT, AIR-PolSAR-Seg, and Sen1Floods11, alongside the EO classification results below. All dense prediction tasks use a frozen encoder and a UPerNet head trained for 10,000 iterations. Common response C1 (parts 1–2) provides the broader comparisons with ten GFMs using their official pretrained checkpoints. All scores below are percentages; bold marks the highest mean within each table column.

(i) Quantitative dense prediction.

BRIGHT: building-damage segmentation. We report mean ± SD over three seeds. SAR is post-event and EO is pre-event. The four class columns report IoU.

ModelInputmIoUBackgroundIntactDamagedDestroyedPixel acc.
SAREO-FMSAR-only43.13 ± 0.4383.46 ± 0.5444.24 ± 0.2419.12 ± 0.8125.72 ± 0.7984.58 ± 0.41
SAREO-FMEO-only (pre-event)52.54 ± 0.3192.50 ± 0.0561.48 ± 0.2022.30 ± 0.3233.89 ± 1.6991.75 ± 0.04
SAREO-FMSAR+EO59.11 ± 0.3092.45 ± 0.0764.60 ± 0.0924.07 ± 0.8855.32 ± 0.5892.52 ± 0.03
SARMAE*SAR-only44.09 ± 0.1584.60 ± 0.0345.49 ± 0.1121.10 ± 0.6625.18 ± 0.3285.59 ± 0.07
SARMAE*EO-only (pre-event)50.81 ± 0.4091.70 ± 0.0759.14 ± 0.3520.12 ± 0.2132.27 ± 1.0691.08 ± 0.11

Joint input improves mIoU by 6.57 points over our stronger unimodal input. EO-only mIoU exceeds SARMAE* by 1.73 points, while SAR-only mIoU is slightly lower (43.13 vs. 44.09).

AIR-PolSAR-Seg: six-class segmentation. This is a SAR-input task. The six class columns report IoU.

Model / readoutmIoUIndustrialNaturalWaterLand useHousingOtherPixel acc.
SAREO-FM (both)54.1840.6473.4078.921.7165.3065.1175.86
SAREO-FM (mod)55.6345.7174.0180.397.3070.4055.9578.72
SARMAE*55.8849.1473.6779.829.5268.8654.2578.67

AIR-PolSAR-Seg: binary water mapping.

Model / readoutmIoUWater IoUNon-water IoUPixel acc.
SAREO-FM (both)89.6480.3698.9198.96
SAREO-FM (mod)89.5280.1498.8998.94
SARMAE*89.4279.9498.9098.94

These results show task-dependent SAR dense transfer: the mod variant is slightly below SARMAE* on six-class mIoU (55.63 vs. 55.88), while our best binary-water mIoU is slightly higher (89.64 vs. 89.42).

Sen1Floods11: flood mapping. We report hand-labelled test-set mean ± SD over seeds 0/1/2. SAR uses S1 VV, and EO uses S2 RGB.

InputmIoUWater IoUNon-water IoUPixel acc.
SAR-only (S1 VV)77.11 ± 0.9860.20 ± 1.7894.02 ± 0.1994.52 ± 0.19
EO-only (S2 RGB)74.19 ± 0.6955.08 ± 1.1893.29 ± 0.2093.80 ± 0.20
SAR+EO82.75 ± 0.5170.07 ± 0.9295.42 ± 0.1195.87 ± 0.11

Joint input improves mIoU by 5.64 points and water IoU by 9.87 points over the stronger unimodal input.

PASTIS evaluation scope. PASTIS [3] uses Sentinel-2 time series with 38–61 acquisitions. Our encoder processes one SAR and/or one EO image without temporal encoding. We therefore focus on BRIGHT, AIR-PolSAR-Seg, and Sen1Floods11 here.

(ii) Additional EO benchmarks.

We added frozen linear-probe evaluations on AID and RESISC45, with EO-only inputs for all methods. Scores are best-epoch mean OA (%). Readout definitions are given in W3; “—” denotes a setting not evaluated. Bold marks the highest mean in each column.

AID.

Method / readout1-shot5-shot10-shot40-shotFull train
SAREO-FM (sem)45.7170.8573.2586.1991.42
SAREO-FM (mod)33.3957.3765.6382.0990.23
SAREO-FM (both)41.6364.7169.6184.9491.97
SARMAE*——49.0569.70—

RESISC45.

Method / readout1-shot5-shot10-shot40-shotFull train
SAREO-FM (sem)34.0259.9266.3672.5188.97
SAREO-FM (mod)26.9348.7658.8269.3088.37
SAREO-FM (both)31.8756.0363.7472.6690.21
SARMAE*——42.0354.36—

Across the available 10- and 40-shot comparisons, sem improves over SARMAE by 16.49–24.33 points*. It is the strongest of our three readouts in seven of eight few-shot settings; both performs best on 40-shot RESISC45 and with the full training split on both benchmarks. These results extend the evidence for EO transfer beyond EuroSAT. We will include these benchmarks and readout comparisons in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer SfKw (3/5)

W3. Data composition, robustness assumptions, and downstream readouts

(i) SAR-1M composition and training modes. SAR-1M contains 1,312,902 SAR observations with various GSDs: 1,042,156 (79.38%) have paired EO observations, and 270,746 (20.62%) are SAR observations without an EO counterpart. Paired/unpaired refers to data availability in the corpus; SAR-only/EO-only/joint refers to the student encoder input. Thus, dropping EO from a paired record produces a SAR-only student input while retaining EO for the teacher.

The table below distinguishes the actual training cases, following Sec. 3.6. The four mode probabilities are conditional on a paired record; loss weights follow Eq. (12). Each retained student image is masked by 75%, while all 256 semantic queries remain active.

Record availabilityStudent input / training modeProbability within paired recordsActive losses
Paired SAR–EOSAR+EO; independent masks0.35Lpixs+Lpixe+Lalign\mathcal{L}_{\mathrm{pix}}^s + \mathcal{L}_{\mathrm{pix}}^e + \mathcal{L}_{\mathrm{align}}
Paired SAR–EOSAR+EO; shared spatial mask0.35Lpixs+Lpixe+Lalign\mathcal{L}_{\mathrm{pix}}^s + \mathcal{L}_{\mathrm{pix}}^e + \mathcal{L}_{\mathrm{align}}
Paired SAR–EOSAR-only; drop_eo0.15Lpixs+Lalign\mathcal{L}_{\mathrm{pix}}^s + \mathcal{L}_{\mathrm{align}}
Paired SAR–EOEO-only; drop_sar0.15Lpixe+Lalign\mathcal{L}_{\mathrm{pix}}^e + \mathcal{L}_{\mathrm{align}}
Unpaired SARSAR-only—Lpixs\mathcal{L}_{\mathrm{pix}}^s only

On paired records, the teacher receives the unmasked EO observation in all four modes, including drop_eo. Reconstruction loss is applied only to retained modalities; a dropped modality has no pixel loss. On unpaired SAR records, no EO observation is available, so both EO reconstruction and semantic alignment are inactive.

The objective also supports unpaired EO through Lpixe+Lalign\mathcal{L}_{\mathrm{pix}}^e + \mathcal{L}_{\mathrm{align}}, using that EO observation as the teacher input. SAR-1M contains no EO-only records, so this case is not used in our pretraining. We will clarify this distinction between the objective's supported cases and the corpus actually used.

(ii) Registration and image-quality assumptions. We use SAR-1M as provided, treating its paired observations as perfectly registered and clean for this study. This is a working assumption, rather than a verified property of every record; we do not explicitly model residual misregistration, cloud contamination, or sensor noise.

We thank the reviewer for highlighting this important research direction. Random relative shifts during pretraining, synthetic cloud/noise injection into EO teacher inputs, and speckle perturbations to SAR would provide useful controlled pretraining studies. Registration robustness is especially important because precisely registered SAR–EO reference pairs are difficult to obtain. Cloud augmentation could also help exploit SAR's ability to observe through clouds when EO is degraded, while sensor-specific noise augmentation would address each modality's own degradations. Pretraining studies with these perturbations and cloud-mask filtering of teacher inputs remain future work; Q1 reports test-time shift and cloud stress tests of the current checkpoint.

(iii) Downstream pooling and readouts. As described in Sec. 3.7 and Appendix E.1, downstream inference passes all available image patches without masking, together with the 256 semantic queries, through the shared encoder; a missing modality's token stream is omitted. The teacher, semantic projector hϕh_\phi, and reconstruction decoders are discarded.

For image-level tasks, sem averages the 256 encoder query outputs into one 768-dimensional vector. These are contextualized encoder outputs, not the initial learnable query embeddings or projected teacher-space features. With joint input, the queries already aggregate SAR and EO information inside the encoder, so sem remains 768-dimensional.

ReadoutPooling and combinationSAR-only or EO-onlyJoint SAR+EO
semMean of all 256 encoder query outputs768768
modMean within each available modality; concatenate the separate SAR and EO means for joint input7681,536
bothConcatenation of mod and sem1,5362,304

The selected vector is passed to the downstream classification head. In frozen linear probing, only that head is trained. We will make these assumptions, training modes, and readout definitions explicit in the revised manuscript.

Reviewer_7NcPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a foundation model for joint SAR and optical representation learning. The key idea is to use a semantic query system that allows (almost) the same shared encoder to process both modalities. The model is trained with an MAE-type objective, which also covers the optical modality. The paper presents extensive experiments on SAR benchmarks, and some on multimodal benchmarks.

Critical strengths
  • The model uses (almost) the same encoder for optical and SAR imagery, which is interesting.
  • A lot of experiments on SAR benchmarks.
  • Relatively clear and well written.
Critical weaknesses
  • The model is very close to the recently published SARMAE (as acknowledged by the authors), and the performance is also very similar. From the reviewer's understanding, the main differences are that the proposed model can process either/and optical or SAR imagery, and that it now includes a reconstruction loss on the optical modality. The work is therefore somewhat derivative, except for its multimodal capability, which is new. However, there is no real comparison with other multimodal models.
  • The paper seems to ignore most of the recent developments in SAR-enabled EO foundation models, both in the related work and in the experimental validation. The presentation of prior work is rather myopic, focusing too much on vision foundation model distillation rather than the wider field of multimodal self-supervised learning for EO. For example, many existing models can process both SAR and optical imagery and are jointly trained on both modalities:
    • Unified streams: AnySat, OlmoEarth, Galileo, Panopticon, SMARTIES
    • Cross-modal MAE: OmniSat, OlmoEarth
    • Unified encoders: UniverSat, RAMEN
  • The masking strategy is not ablated, and previous work on masking, such as PRESTO, is not mentioned as prior exploration of the topic.
  • The paper claims to introduce a foundation model, but there is no mention of code or pretrained model release.
  • It is not exactly a shared encoder, since the model still relies on modality-specific linear projections (minor).

Overall:

This work improves on the recent SARMAE with an added loss and a multimodal stream. The approach remains very close to SARMAE, both in concept and performance. The main novelty is its multimodal capability, which is a meaningful extension. But the related work and experiments ignore most of the developments in multimodal EO foundation models and self-supervised learning over the last four years. If the paper aims to introduce a multimodal foundation model, it needs to compare against existing multimodal foundation models.

[1] Astruc et al., AnySat: One Earth Observation Model for Many Resolutions, Scales, and Modalities, CVPR 2025.
[2] Herzog et al., OlmoEarth: Stable Latent Image Modeling for Multimodal Earth Observation, CVPR 2026
[3] Tseng et al., Galileo: Learning Global and Local Features in Pretrained Remote Sensing Models, ICML 2025
[4] Waldmann et al., Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation, CVPR Workshops 2025
[5] Sumbul et al., SMARTIES: Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images, ICCV 2025
[6] Astruc et al., OmniSat: Self-Supervised Modality Fusion for Earth Observation, ECCV 2024
[7] Perron et al., UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation, NeurIPS 2026
[8] Houdré et al., RAMEN: Resolution-adjustable multimodal encoder for Earth observation, CVPR 2026
[9] Tseng et al., PRESTO; Lightweight, pre-trained transformers for remote sensing timeseries arXiv 2023

Questions for authors
  • What is meant by a "clean and unmasked observation"? In particular, what does "clean" mean here?

  • What is a "supervision head"? How is it semantic? More generally, why are the queries qualified as semantic?

  • It is very unclear how multiple spatial resolutions (from 0.1 to 60 m!) are handled by a single SAR encoder.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7NcP (1/2)

We sincerely thank the reviewer for carefully reading our manuscript and providing detailed, constructive feedback. Your comments have helped us clarify the contribution beyond SARMAE and better position our work within the broader multimodal EO literature.

W1. Difference from SARMAE

We agree that SARMAE was our starting point. SAREO-FM introduces two key changes (Fig. 1):

(i) Decoupled supervision. Semantic alignment targets learnable queries, while reconstruction targets modality tokens. The queries interact with image tokens but are excluded from the reconstruction decoders.

(ii) Support for varying sensor availability. Raw EO encoding and mixed modality masking enable one pretrained encoder to support SAR-only, EO-only, and joint inputs.

The expanded comparison in Common response C1 (parts 1–2) shows higher mean OA than SARMAE* on FUSAR-Ship and SAR-ACD at both 10 and 40 shots. The multimodal comparisons and their scope are addressed below.

W2. Multimodal foundation models and related work

Our original evaluation, like SARMAE's, emphasized SAR transfer. We thank the reviewer for pointing us to the broader multimodal EO literature. We will expand the related work to cover all nine suggested references.

We have now added comparisons with CROMA, SMARTIES, Galileo, RAMEN, UniverSat, DOFA, Panopticon, AnySat, PRESTO, and OlmoEarth. All ten baselines use the official pretrained checkpoints released by their authors and are evaluated with frozen encoders. The full results, including our semantic-query and modality-token readouts on SAR benchmarks, are posted in Common response C1 (parts 1–2).

SAR transfer at GSDs around 1 m. Our semantic-query readout obtains the highest mean OA among the compared models on both FUSAR-Ship (81.65/89.56) and SAR-ACD (53.65/71.72) at 10/40 shots. The gains are significant against many GFMs; the small 10-shot SAR-ACD margin over CROMA is a numerical gain.

Sentinel transfer at 10 m GSD. Our EO-only and joint scores on BigEarthNet-MM (68.30/68.90 micro-AP) rank second behind OlmoEarth. On So2Sat, OlmoEarth, Galileo, and Panopticon remain stronger in these settings. Thus, the expanded comparison shows a clearer advantage on FUSAR-Ship and SAR-ACD, with task-dependent EO and joint transfer.

Pretraining scope. SAR-1M contains 1,312,902 SAR observations with various GSDs: 79.38% have geographically corresponding EO observations, and 20.62% are SAR-only. This SAR-centered corpus motivated our initial evaluation emphasis. Modality masking also exposes the encoder to EO-only and joint inputs using the paired observations. We will clarify this scope in the manuscript.

We will discuss OmniSat [6] in the related work. It was not evaluated because no general-purpose checkpoint was available; its pretraining uses each target dataset's training split. C1 also explains the hyperspectral input requirements preventing a direct comparison with SpectralEarth-FM. We will include the expanded comparisons and their scope in the revised manuscript.

W3. Masking strategy and prior work

We will cite PRESTO [9], OmniSat [6], and OlmoEarth [2] and discuss their masking studies: band/time masking, independent/shared/whole-modality masking, and modality-aware band masking, respectively. Our training combines independent masking, shared spatial masking, EO dropping, and SAR dropping to support varying sensor availability.

The table below reports a masking ablation, separately from the checkpoint comparisons in C1. Scores are OA (%); Avg. is the unweighted mean across the three benchmarks, and Δ is the difference in Avg. from the full configuration.

ConfigurationMSTARFUSAR-ShipSAR-ACDAvg.Δ
Full configuration (reference)78.4184.0052.0971.500.00
No modality dropping44.9949.3528.3940.91−30.59
No shared spatial masking76.8582.5651.3070.24−1.26

These ablations show a large contribution from modality dropping and a smaller, consistent benefit from shared spatial masking.

W4. Code and model release

We will release the code, pretrained encoder, and exact pretraining configuration to make the results reproducible.

W5. Shared encoder terminology

Although the patch embeddings are modality-specific, 98.4% of the inference-encoder parameters are shared. We will use the more precise description “a shared Transformer encoder with modality-specific patch embeddings.”

AuthorsOpenReview ↗

Response to Reviewer 7NcP (2/2)

Q1. Meaning of “clean”

By “clean,” we meant unmasked, referring to the absence of patch masking. We will consistently use “unmasked EO observation” to avoid confusion.

Q2. Supervision head and semantic queries

The supervision head is the lightweight semantic projector hϕh_\phi, which maps query outputs into the frozen VFM teacher's feature space. The queries are called semantic because their outputs are explicitly trained to match the teacher's high-level visual representations through the alignment loss. The term describes their training objective; the projector supplies the mapping needed to apply that objective.

The feature-map visualization compares teacher features, SAR-token features, and query features under SAR-only, EO-only, and joint inputs. It illustrates the coarse scene-region structure learned through this teacher guidance. (Figure 6 in Appendix)

Q3. Handling different ground resolutions

The 0.1–60 m range describes GSD; the encoder always receives 256 × 256 pixels. For the 89% of images already at this size, we crop on the original pixel grid (area 50–100%, aspect ratio 3/4–4/3) and bicubically resize to 256 × 256. The remaining 11%, with images ≥128 × 128, are resized to 256 × 256. This standardizes input dimensions while physical ground scale varies. We use no GSD embedding and will clarify this limitation; explicit GSD conditioning is a useful future direction.

Reviewer_7NcPOpenReview ↗

Post rebuttal

The reviewer thanks the authors for their thorough rebuttal and the substantial additional experimental effort. In particular, the expanded comparison with existing multimodal EO foundation models satisfactorily addresses the reviewer's main concern. The additional masking ablation, methodological clarifications, and commitment to releasing the code and pretrained model are also appreciated.

While the methodological contribution remains somewhat incremental with respect to SARMAE, the expanded evaluation better establishes the relevance of the proposed model within the broader landscape of EO foundation models.

The reviewer no longer has major objections to the paper and is therefore raising their recommendation to weak acceptance.

We sincerely thank the reviewer for the constructive feedback and for recognizing our efforts. The suggestions helped us strengthen the experimental comparison and improve the clarity of the paper. We will incorporate the additional results and clarifications into the revised manuscript and release the code and pretrained model as promised.

Reviewer_qBhvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SAREO-FM, a self-supervised foundation model supporting synthetic aperture radar (SAR), electro-optical (EO), and joint SAR–EO inputs. Its central design introduces learnable semantic queries into a shared encoder: the queries receive semantic supervision from a frozen DINOv3 teacher, while modality tokens support masked pixel reconstruction, separating the direct targets of the two objectives. Independent masking, shared spatial masking, and modality dropping accommodate different input configurations. The model is pretrained on SAR-1M and evaluated through linear probing and fine-tuning across SAR/EO classification and paired single-label and multi-label recognition benchmarks. Validation also includes cumulative component ablations and qualitative building-damage assessment on BRIGHT.

Critical strengths
  1. The paper presents a well-motivated design that assigns semantic distillation to dedicated queries and pixel reconstruction to modality tokens while preserving information exchange within a shared encoder. This turns the challenge of coordinating learning objectives across heterogeneous sensors into a concrete, testable architectural choice.

  2. A single pretrained encoder supports SAR-only, EO-only, and joint SAR–EO inputs, without requiring the teacher or reconstruction decoders during downstream inference. This unified interface has practical value for remote-sensing applications with varying sensor availability.

Critical weaknesses

1.Table 4 attributes a 3.13% average improvement to decoupled supervision, but this modification also introduces 256 semantic queries, changing the encoder’s sequence length and information interactions. Matching training steps and batch size does not control for the additional computation, and a matched variant retaining the same queries while supervising modality tokens is missing. Consequently, the ablation does not establish whether changing the supervision destination drives the improvement, leaving the paper’s central methodological contribution insufficiently validated.

2.Reevaluating SARMAE* under a common downstream protocol provides a useful model comparison, but it does not isolate the effects of raw EO inputs and cross-modal learning as claimed. The proposed model uses a DINOv3-7B teacher, while matching teacher configuration, initialization, augmentation, and pretraining budget across the compared checkpoints is not sufficiently established. The observed differences therefore cannot be cleanly attributed to the proposed design. A direct-alignment baseline using the same teacher and training recipe is needed to substantiate the claimed source of improvement.

Questions for authors

1.Between the direct DINOv3 supervision and decoupled supervision variants in Table 4, are the query count, spatial locations receiving semantic supervision, and loss normalization matched? Do you have results for a control that retains the same queries but applies semantic supervision to modality tokens?

2.On So2Sat and BigEarthNet-MM, does joint input still outperform unimodal input when all three configurations use the fixed 768-dimensional sem readout? Does joint encoding also improve over simple late fusion of separately encoded SAR and EO features?

1 comment
AuthorsOpenReview ↗

Response to Reviewer qBhv

We thank the reviewer for identifying the controls needed to assess decoupled supervision and multimodal transfer.

W1 / Q1. Query-retaining controls and supervision locations

We agree that adding queries and changing the destination of semantic supervision require separate controls. We added the requested variant that retains semantic queries while applying DINOv3 supervision to SAR tokens, followed by query supervision at the same 64 visible SAR positions and over the full 256-position grid.

The table reports OA (%).

Encoder and alignment configurationMSTARFUSARSAR-ACDAvg.Δ
SAR tokens; supervise SAR tokens73.4979.9048.6267.340.00
SAR + query tokens; supervise SAR tokens73.8680.2550.3368.15+0.81
SAR + query tokens; supervise queries at 64 visible SAR positions76.2281.6250.6369.49+1.34
Decoupled: supervise queries at all 256 positions77.1182.5551.0470.23+0.74

Adding queries while retaining SAR-token supervision improves average OA by 0.81 points. Redirecting supervision to queries at the same 64 spatial positions adds 1.34 points; extending query supervision to all 256 positions adds another 0.74 points. This pattern supports a role for supervision placement and spatial coverage beyond query addition.

Q1 clarification. The added query-retaining and 64-position controls separate query addition, supervision placement, and spatial coverage. All variants average the alignment loss over the supervised spatial positions, with N=64N=64 for the 64-position controls and N=256N=256 for full-grid supervision. We will explicitly document this normalization, query count, alignment weight, and training budget.

W2. Teacher scale versus supervision design

We agree that evaluating the official SARMAE checkpoint with the same downstream protocol does not isolate pretraining design choices. The controls in W1/Q1 examine query addition and supervision placement within our framework. We also evaluate SAREO-FM with DINOv3-7B, DINOv3-L, and DINOv3-B, the teacher size used by SARMAE.

The table reports OA (%); All rows are SAREO-FM variants.

TeacherMSTARFUSARSAR-ACDAvg.Δ
DINOv3-7B (Ours)78.4184.0052.0971.500.00
DINOv3-L78.6383.1252.2171.32−0.18
DINOv3-B (SARMAE teacher size)77.6183.5752.3771.18−0.14

The observed variation across teacher sizes is smaller than the gains from learnable queries and supervision placement in W1/Q1: average OA varies by 0.32 points across teachers, compared with the 2.90-point difference across the query/supervision variants. This suggests that the larger teacher alone does not explain the observed gains. We will document teacher configuration, initialization, augmentation, and training budget for each variant, without describing the separate SARMAE checkpoint comparison as an isolated test of raw EO inclusion.

Q2. Fixed-dimensional readouts and late fusion

Thank you for suggesting this fixed-dimensional control. It helped us identify a query-only readout that gives higher EO-only and joint scores than those reported in Table 3. Table 3 used modality-token readouts for unimodal inputs and a combined modality/query readout for joint inputs; it omitted query-only results. We will add these results and specify the feature choice consistently.

The table reports best-epoch scores from the main checkpoint: So2Sat OA (%) and BigEarthNet-MM micro-AP (%). The semantic-query scores are the same ones used in Common response C1 (part 2).

sem is the mean of the 256 encoder query outputs, without the semantic projector; mod pools modality-token outputs. Joint encodes SAR and EO together. Late mean and Late concat. combine features from separate SAR/EO encoding by averaging and concatenation, respectively.

InputFeaturesEncoding / fusionDim.So2Sat OABigEarthNet-MM micro-AP
SARsemSingle76829.5461.35
EOsemSingle76859.7568.30
SAR+EOsemJoint76860.2468.90
SAR+EOsemLate mean76857.4568.85
SAR+EOmodJoint1,53656.4765.02
SAR+EOmodLate concat.1,53655.0065.31
SAR+EOmod + semJoint2,30458.7966.70

With fixed 768-dimensional features, joint input exceeds the stronger unimodal input by 0.49 points on So2Sat and 0.60 on BigEarthNet-MM. Against mean-feature late fusion, joint encoding improves So2Sat by 2.79 points, while BigEarthNet-MM is nearly tied (68.90 vs. 68.85). Modality-token results are also task-dependent: joint encoding exceeds late concatenation on So2Sat but is slightly lower on BigEarthNet-MM. We will scope the fusion claim to these measured differences.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Common Response to All Reviewers

We thank all reviewers for their careful reading and constructive feedback, and for highlighting our well-motivated design, multimodal flexibility, and clear presentation.

We apologize for the delay in responding while completing extensive additional experiments to address the reviewers’ concerns. We will post a detailed, individual response to each reviewer within the next 24 hours.

AuthorsOpenReview ↗

Common Response to All Reviewers (1/2)

C1. Broader comparisons with geospatial foundation models (GFMs) (1/2)

We thank the reviewers for encouraging broader comparisons with geospatial foundation models (GFMs). We have expanded the evaluation to include CROMA [1], SMARTIES [2], Galileo [3], RAMEN [4], UniverSat [5], DOFA [6], Panopticon [7], AnySat [8], PRESTO [9], and OlmoEarth [10], alongside the SAR-specific SARMAE* [11] reference. For all ten GFM baselines, we used the official pretrained checkpoints released by the authors.

SAR transfer at fine ground resolutions. Our semantic-query readout achieves the highest mean OA on FUSAR-Ship and SAR-ACD (both with GSDs around 1 m) at both 10 and 40 shots, with statistically significant gains over many GFMs. The gains over the strongest multimodal GFM in each setting are +2.67/+1.58 points on FUSAR-Ship and +0.58/+1.76 points on SAR-ACD at 10/40 shots. The small 10-shot SAR-ACD margin over CROMA is reported as a numerical gain.

Scores are best-epoch mean OA (%) over 10 identical support draws with frozen encoders. Both our semantic-query (sem) and modality-token (mod) readouts are shown. Bold and underline mark the highest and second-highest means in each column.

Model / readoutFUSAR-Ship 10-shotFUSAR-Ship 40-shotSAR-ACD 10-shotSAR-ACD 40-shot
SAREO-FM (sem)81.6589.5653.6571.72
SAREO-FM (mod)76.5488.97‾\underline{88.97}50.8769.39
SARMAE* [11]77.6888.3451.6868.98
CROMA [1]74.5583.8353.07‾\underline{53.07}67.54
SMARTIES [2]65.2779.5448.0761.18
Galileo [3]70.8683.2051.9669.06
RAMEN [4]59.6075.9042.3048.84
UniverSat [5]56.8172.8432.9138.11
DOFA [6]77.2787.0350.5469.96‾\underline{69.96}
Panopticon [7]78.98‾\underline{78.98}87.9849.9968.48
AnySat [8]57.5970.9244.9356.89
PRESTO [9]51.1168.4527.4128.39
OlmoEarth [10]73.8284.5650.3266.44
AuthorsOpenReview ↗

Common Response to All Reviewers (2/2)

Common response: Broader comparisons with geospatial foundation models (GFMs) (2/2)

Sentinel transfer at 10 m GSD. Our EO results are task-dependent. On BigEarthNet-MM, our semantic-query readout achieves the second-highest mean scores with EO-only and joint inputs (68.30/68.90), behind OlmoEarth. On So2Sat, OlmoEarth, Galileo, and Panopticon remain stronger in these settings. These results show useful EO and multimodal transfer, with a clearer advantage on FUSAR-Ship and SAR-ACD.

For a SAR-specific reference, SARMAE* obtains 57.44 on BigEarthNet-MM and 28.96 on So2Sat with SAR-only input, compared with 61.35 and 29.54 from our semantic-query readout. The So2Sat gain is modest, further indicating that our SAR advantage depends on the benchmark. SAREO-FM additionally supports EO-only and joint inference with the same encoder.

The table below reports best-epoch means with frozen encoders: BigEarthNet-MM (BEN), micro-AP (%) over 5 support draws; So2Sat, OA (%) over 10 support draws. Both benchmarks use 10 m GSD. SAR denotes S1, and EO denotes S2. SAREO-FM always uses VV for SAR and RGB (B04/B03/B02) for EO, with the semantic-query readout. Each baseline uses an input configuration supported by its released checkpoint. Bold and underline mark the highest and second-highest means within each column; a dash denotes an unsupported input configuration.

Sentinel benchmark comparisons

Model / readoutBEN SARBEN EOBEN SAR+EOSo2Sat SARSo2Sat EOSo2Sat SAR+EO
SAREO-FM (sem)61.35‾\underline{61.35}68.30‾\underline{68.30}68.90‾\underline{68.90}29.5459.7560.24
SARMAE* [11]57.44——28.96——
CROMA [1]58.3064.1967.2429.89——
SMARTIES [2]57.3554.9158.3426.9644.2143.05
Galileo [3]57.3860.1163.6029.4163.87‾\underline{63.87}63.31‾\underline{63.31}
RAMEN [4]52.4758.1459.1326.3844.7245.18
UniverSat [5]52.9455.3158.5122.7240.6543.02
DOFA [6]56.7163.7461.4124.8854.6751.62
Panopticon [7]63.0068.0867.1631.48‾\underline{31.48}63.0262.33
AnySat [8]51.8962.2962.9123.0148.6748.48
PRESTO [9]52.3252.6457.5924.5837.1845.69
OlmoEarth [10]59.1172.3572.5531.5870.4169.56

Models not evaluated. We could not include OmniSat [12] because no general-purpose checkpoint was available (pretraining uses each target dataset’s training split). SpectralEarth-FM [13] requires at least 75 spectral bands, exceeding those available in our SAR/Sentinel inputs.

Scope and interpretation. Our emphasis on SAR follows the composition and purpose of SAR-1M: it is assembled around SAR observations, with EO available for a subset of them. Modality masking makes this corpus usable for EO-only and joint training, but does not expand its EO coverage or spectral bands. The results support a SAR-centered foundation model that also enables EO and joint transfer, with particularly strong performance on FUSAR-Ship and SAR-ACD. We will clarify this scope and include the expanded comparisons in the revised manuscript.

References

[1] Anthony Fuller, Koreen Millard, and James R. Green. CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders. NeurIPS 2023.

[2] Gencer Sumbul et al. SMARTIES: Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images. ICCV 2025.

[3] Gabriel Tseng et al. Galileo: Learning Global & Local Features of Many Remote Sensing Modalities. ICML 2025.

[4] Nicolas Houdré et al. RAMEN: Resolution-Adjustable Multimodal Encoder for Earth Observation. CVPR 2026.

[5] Yohann Perron et al. UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation. NeurIPS 2026.

[6] Zhitong Xiong et al. Neural Plasticity-Inspired Foundation Model for Observing the Earth Crossing Modalities. arXiv:2403.15356, 2024.

[7] Leonard Waldmann et al. Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation. CVPR Workshops 2025.

[8] Guillaume Astruc et al. AnySat: One Earth Observation Model for Many Resolutions, Scales, and Modalities. CVPR 2025.

[9] Gabriel Tseng et al. Lightweight, Pre-trained Transformers for Remote Sensing Timeseries. NeurIPS Workshop on Tackling Climate Change with Machine Learning, 2023.

[10] Henry Herzog et al. OlmoEarth: Stable Latent Image Modeling for Multimodal Earth Observation. CVPR 2026.

[11] Danxu Liu et al. SARMAE: Masked Autoencoder for SAR Representation Learning. CVPR 2026.

[12] Guillaume Astruc et al. OmniSat: Self-Supervised Modality Fusion for Earth Observation. ECCV 2024.

[13] Nassim Ait Ali Braham et al. SpectralEarth: Training Hyperspectral Foundation Models at Scale. IEEE JSTARS, 2025.

AuthorsOpenReview ↗

Revised manuscript and follow-up

Dear reviewers and AC,

Thank you for your thoughtful feedback and discussion. We have revised the manuscript to incorporate the additional experiments and clarifications from our responses. The main paper now compares SAREO-FM with ten multimodal geospatial foundation models and highlights its strongest comparative results on fine-resolution SAR benchmarks. The appendix adds quantitative dense prediction, additional EO classification benchmarks, tests under EO misregistration and cloud occlusion, and controlled studies of modality masking, semantic supervision, teacher scale, and late fusion.

If you have any remaining questions about the revision or our responses, we would be happy to clarify. Thank you again for your time and feedback.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.