- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a unified SAR–EO foundation model that decouples high-level semantic alignment from modality-specific reconstruction within a single masked autoencoding framework. The authors introduce learnable, spatially organized semantic queries that are aligned patch-wise to features from a frozen EO vision FM (DINOv3), while SAR and EO image tokens are used exclusively for masked pixel reconstruction. All three streams interact in a shared encoder.
Critical strengths
The authors address a practically important challenge in geospatial foundation modeling: learning modality-specific details and cross-modal semantics without conflict
-
As technical novelty, the authors propose a decoupled semantic supervision via persistent learnable queries. This can be considered as a relevant answer to the “one stream, two duties” issue common in MAE-style multimodal pretraining. In addition, the unified three-stream encoder (SAR tokens, EO tokens, and semantic queries) together with modality-specific decoders and query-only alignment is a principled architectural separation that preserves cross-stream interaction while preventing supervision entanglement. This seems novel to me. The mixed modality masking strategy is well designed to expose the same encoder to SAR-only, EO-only, and paired inputs. This may improve the robustness to missing modalities.
-
Experiments include fair controlled comparison against SARMAE with matched backbones, datasets, and protocols isolating the effect of raw EO inclusion and the decoupled objective. Evaluations span SAR-only, EO-only , and joint SAR–EO , plus a qualitative dense-prediction example on BRIGHT. The ablation study is fair enough to show the individual contribution of each component (MAE, DINOv3 supervision, decoupled supervision, raw EO modality, etc.)
-
The motivation is well-articulated, and the method is presented with clear mathematical detail, enabling reproducibility.
Critical weaknesses
There are some technical limitations or concerns:
-
From a methodological point of view, there is a big dependence on a large frozen EO teacher (DINOv3) and on paired SAR–EO data for alignment. This may raise questions about scalability to domains with limited or misregistered EO partners, and about potential optical bias in the learned semantics. Also the semantic queries are fixed in count and unmasked. The authors did not explore sensitivity to query number, masking, or alternative alignment destinations (e.g., a small set of global queries vs. full patch grid).
-
From an experimental point of view, quantitative dense prediction results are quite limited. Only qualitative BRIGHT examples are shown, leaving open how well the approach transfers to pixel-level tasks where SAR excels. Comparisons to broader multimodal GFMs (e.g., OmniSat, RAMEN, SpectralEarth-FM) are not reported, even if not strictly SAR-focused, which would better contextualize the contribution within current EO multimodal pretraining trends. EO transfer is evaluated mainly on EuroSAT. Additional EO benchmarks (e.g., RESISC45, AID in the main results) are mentioned but not shown in tables, limiting the EO-branch evidence...
-
For clarity, some training-mode details (e.g., exact share of unpaired samples, handling of imperfect coregistration, and cloud/noise effects in teacher) could be made more explicit in the paper. The downstream usage of semantic-query features (pooling, readout strategy) could be specified more clearly for reproducibility.
-
For positioning in the SOA, the paper would benefit from deeper discussion and experimental contrast to any-to-any and resolution-aware multimodal MAE/JEPA paradigms (e.g., MultiMAE/4M, OmniSat, RAMEN) that also address missing modalities and decoders per modality.
Questions for authors
How robust is the approach to SAR–EO misregistration and cloud contamination in teacher features? Can you report performance under synthetic misalignment or cloud‑mask filtering of teacher inputs? How does the method perform on quantitative dense prediction tasks (e.g., Sen1Floods11, PASTIS, BRIGHT) with SAR-only, EO-only, and joint inputs?
5 comments
Response to Reviewer SfKw (2/5)
W2. Quantitative dense prediction and additional EO benchmarks
We added quantitative dense prediction on BRIGHT, AIR-PolSAR-Seg, and Sen1Floods11, alongside the EO classification results below. All dense prediction tasks use a frozen encoder and a UPerNet head trained for 10,000 iterations. Common response C1 (parts 1–2) provides the broader comparisons with ten GFMs using their official pretrained checkpoints. All scores below are percentages; bold marks the highest mean within each table column.
(i) Quantitative dense prediction.
BRIGHT: building-damage segmentation. We report mean ± SD over three seeds. SAR is post-event and EO is pre-event. The four class columns report IoU.
| Model | Input | mIoU | Background | Intact | Damaged | Destroyed | Pixel acc. |
|---|---|---|---|---|---|---|---|
| SAREO-FM | SAR-only | 43.13 ± 0.43 | 83.46 ± 0.54 | 44.24 ± 0.24 | 19.12 ± 0.81 | 25.72 ± 0.79 | 84.58 ± 0.41 |
| SAREO-FM | EO-only (pre-event) | 52.54 ± 0.31 | 92.50 ± 0.05 | 61.48 ± 0.20 | 22.30 ± 0.32 | 33.89 ± 1.69 | 91.75 ± 0.04 |
| SAREO-FM | SAR+EO | 59.11 ± 0.30 | 92.45 ± 0.07 | 64.60 ± 0.09 | 24.07 ± 0.88 | 55.32 ± 0.58 | 92.52 ± 0.03 |
| SARMAE* | SAR-only | 44.09 ± 0.15 | 84.60 ± 0.03 | 45.49 ± 0.11 | 21.10 ± 0.66 | 25.18 ± 0.32 | 85.59 ± 0.07 |
| SARMAE* | EO-only (pre-event) | 50.81 ± 0.40 | 91.70 ± 0.07 | 59.14 ± 0.35 | 20.12 ± 0.21 | 32.27 ± 1.06 | 91.08 ± 0.11 |
Joint input improves mIoU by 6.57 points over our stronger unimodal input. EO-only mIoU exceeds SARMAE* by 1.73 points, while SAR-only mIoU is slightly lower (43.13 vs. 44.09).
AIR-PolSAR-Seg: six-class segmentation. This is a SAR-input task. The six class columns report IoU.
| Model / readout | mIoU | Industrial | Natural | Water | Land use | Housing | Other | Pixel acc. |
|---|---|---|---|---|---|---|---|---|
| SAREO-FM (both) | 54.18 | 40.64 | 73.40 | 78.92 | 1.71 | 65.30 | 65.11 | 75.86 |
| SAREO-FM (mod) | 55.63 | 45.71 | 74.01 | 80.39 | 7.30 | 70.40 | 55.95 | 78.72 |
| SARMAE* | 55.88 | 49.14 | 73.67 | 79.82 | 9.52 | 68.86 | 54.25 | 78.67 |
AIR-PolSAR-Seg: binary water mapping.
| Model / readout | mIoU | Water IoU | Non-water IoU | Pixel acc. |
|---|---|---|---|---|
| SAREO-FM (both) | 89.64 | 80.36 | 98.91 | 98.96 |
| SAREO-FM (mod) | 89.52 | 80.14 | 98.89 | 98.94 |
| SARMAE* | 89.42 | 79.94 | 98.90 | 98.94 |
These results show task-dependent SAR dense transfer: the mod variant is slightly below SARMAE* on six-class mIoU (55.63 vs. 55.88), while our best binary-water mIoU is slightly higher (89.64 vs. 89.42).
Sen1Floods11: flood mapping. We report hand-labelled test-set mean ± SD over seeds 0/1/2. SAR uses S1 VV, and EO uses S2 RGB.
| Input | mIoU | Water IoU | Non-water IoU | Pixel acc. |
|---|---|---|---|---|
| SAR-only (S1 VV) | 77.11 ± 0.98 | 60.20 ± 1.78 | 94.02 ± 0.19 | 94.52 ± 0.19 |
| EO-only (S2 RGB) | 74.19 ± 0.69 | 55.08 ± 1.18 | 93.29 ± 0.20 | 93.80 ± 0.20 |
| SAR+EO | 82.75 ± 0.51 | 70.07 ± 0.92 | 95.42 ± 0.11 | 95.87 ± 0.11 |
Joint input improves mIoU by 5.64 points and water IoU by 9.87 points over the stronger unimodal input.
PASTIS evaluation scope. PASTIS [3] uses Sentinel-2 time series with 38–61 acquisitions. Our encoder processes one SAR and/or one EO image without temporal encoding. We therefore focus on BRIGHT, AIR-PolSAR-Seg, and Sen1Floods11 here.
(ii) Additional EO benchmarks.
We added frozen linear-probe evaluations on AID and RESISC45, with EO-only inputs for all methods. Scores are best-epoch mean OA (%). Readout definitions are given in W3; “—” denotes a setting not evaluated. Bold marks the highest mean in each column.
AID.
| Method / readout | 1-shot | 5-shot | 10-shot | 40-shot | Full train |
|---|---|---|---|---|---|
| SAREO-FM (sem) | 45.71 | 70.85 | 73.25 | 86.19 | 91.42 |
| SAREO-FM (mod) | 33.39 | 57.37 | 65.63 | 82.09 | 90.23 |
| SAREO-FM (both) | 41.63 | 64.71 | 69.61 | 84.94 | 91.97 |
| SARMAE* | — | — | 49.05 | 69.70 | — |
RESISC45.
| Method / readout | 1-shot | 5-shot | 10-shot | 40-shot | Full train |
|---|---|---|---|---|---|
| SAREO-FM (sem) | 34.02 | 59.92 | 66.36 | 72.51 | 88.97 |
| SAREO-FM (mod) | 26.93 | 48.76 | 58.82 | 69.30 | 88.37 |
| SAREO-FM (both) | 31.87 | 56.03 | 63.74 | 72.66 | 90.21 |
| SARMAE* | — | — | 42.03 | 54.36 | — |
Across the available 10- and 40-shot comparisons, sem improves over SARMAE by 16.49–24.33 points*. It is the strongest of our three readouts in seven of eight few-shot settings; both performs best on 40-shot RESISC45 and with the full training split on both benchmarks. These results extend the evidence for EO transfer beyond EuroSAT. We will include these benchmarks and readout comparisons in the revised manuscript.
Response to Reviewer SfKw (3/5)
W3. Data composition, robustness assumptions, and downstream readouts
(i) SAR-1M composition and training modes. SAR-1M contains 1,312,902 SAR observations with various GSDs: 1,042,156 (79.38%) have paired EO observations, and 270,746 (20.62%) are SAR observations without an EO counterpart. Paired/unpaired refers to data availability in the corpus; SAR-only/EO-only/joint refers to the student encoder input. Thus, dropping EO from a paired record produces a SAR-only student input while retaining EO for the teacher.
The table below distinguishes the actual training cases, following Sec. 3.6. The four mode probabilities are conditional on a paired record; loss weights follow Eq. (12). Each retained student image is masked by 75%, while all 256 semantic queries remain active.
| Record availability | Student input / training mode | Probability within paired records | Active losses |
|---|---|---|---|
| Paired SAR–EO | SAR+EO; independent masks | 0.35 | |
| Paired SAR–EO | SAR+EO; shared spatial mask | 0.35 | |
| Paired SAR–EO | SAR-only; drop_eo | 0.15 | |
| Paired SAR–EO | EO-only; drop_sar | 0.15 | |
| Unpaired SAR | SAR-only | — | only |
On paired records, the teacher receives the unmasked EO observation in all four modes, including drop_eo. Reconstruction loss is applied only to retained modalities; a dropped modality has no pixel loss. On unpaired SAR records, no EO observation is available, so both EO reconstruction and semantic alignment are inactive.
The objective also supports unpaired EO through , using that EO observation as the teacher input. SAR-1M contains no EO-only records, so this case is not used in our pretraining. We will clarify this distinction between the objective's supported cases and the corpus actually used.
(ii) Registration and image-quality assumptions. We use SAR-1M as provided, treating its paired observations as perfectly registered and clean for this study. This is a working assumption, rather than a verified property of every record; we do not explicitly model residual misregistration, cloud contamination, or sensor noise.
We thank the reviewer for highlighting this important research direction. Random relative shifts during pretraining, synthetic cloud/noise injection into EO teacher inputs, and speckle perturbations to SAR would provide useful controlled pretraining studies. Registration robustness is especially important because precisely registered SAR–EO reference pairs are difficult to obtain. Cloud augmentation could also help exploit SAR's ability to observe through clouds when EO is degraded, while sensor-specific noise augmentation would address each modality's own degradations. Pretraining studies with these perturbations and cloud-mask filtering of teacher inputs remain future work; Q1 reports test-time shift and cloud stress tests of the current checkpoint.
(iii) Downstream pooling and readouts. As described in Sec. 3.7 and Appendix E.1, downstream inference passes all available image patches without masking, together with the 256 semantic queries, through the shared encoder; a missing modality's token stream is omitted. The teacher, semantic projector , and reconstruction decoders are discarded.
For image-level tasks, sem averages the 256 encoder query outputs into one 768-dimensional vector. These are contextualized encoder outputs, not the initial learnable query embeddings or projected teacher-space features. With joint input, the queries already aggregate SAR and EO information inside the encoder, so sem remains 768-dimensional.
| Readout | Pooling and combination | SAR-only or EO-only | Joint SAR+EO |
|---|---|---|---|
| sem | Mean of all 256 encoder query outputs | 768 | 768 |
| mod | Mean within each available modality; concatenate the separate SAR and EO means for joint input | 768 | 1,536 |
| both | Concatenation of mod and sem | 1,536 | 2,304 |
The selected vector is passed to the downstream classification head. In frozen linear probing, only that head is trained. We will make these assumptions, training modes, and readout definitions explicit in the revised manuscript.
Response to Reviewer SfKw (1/5)
We sincerely thank the reviewer for the careful reading and thoughtful, constructive feedback. Your suggestions have helped us strengthen our evaluations of dense prediction and EO transfer, clarify the training and readout protocols, and better explain the robustness limitations and relation to prior multimodal work.
W1. Teacher dependence, modality masking, and query design
(i) Teacher dependence. As reported in our response to Reviewer qBhv, W2, DINOv3-7B/L/B yield average OA of 71.47/71.32/71.18, a range of only 0.32 points. This is smaller than the 2.90-point gain across the query/supervision variants in qBhv W1/Q1, suggesting that our gains are not primarily explained by teacher scale.
(ii) Modality masking. The ablation in our response to Reviewer 7NcP, W3 gives average OA of 71.50 for the full configuration, 40.91 without modality dropping (−30.59 points), and 70.24 without shared spatial masking (−1.26 points). These results support explicitly training for varying sensor availability, with modality dropping making the largest contribution.
(iii) Alignment destination and query granularity. The SAR-only controls in Reviewer qBhv, W1/Q1 show successive average OA gains of +0.81 points from adding queries, +1.34 from redirecting alignment to queries at the same 64 positions, and +1.07 from extending alignment to all 256 positions. All variants average the alignment loss over their supervised spatial positions. These controls support both the supervision destination and spatial coverage.
We additionally evaluate 4×4 (16-query) and 1×1 (global-query) pooled configurations in the SAR+EO ablation, compared with the full 16×16 query grid. Scores are OA (%); Avg. is the unweighted mean across benchmarks, and Δ is the change in Avg. from the preceding row, in percentage points.
Pooling to 4×4 and 1×1 lowers average OA by 10.83 and 13.36 points relative to the full grid. This suggests that mixing features from different regions weakens the local information needed for semantic alignment. All queries remain unmasked.