- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a self-supervised pretraining framework for multimodal Earth observation foundation models called ScalSpectra. The goal is to build a unified model that can work with different spectral configurations, including RGB, multispectral and SAR, etc. observations, while also learning good dense patch representations.
A main component is SADBT, which uses physical band descriptors and a configuration-aware query to fuse the available bands. The important idea is that the contribution of a band depends not only on the band itself, but also on which other bands are observed together with it. The framework further uses dual-teacher dense feature learning to improve the local and semantic quality of the patch features. The model is evaluated on scene classification, semantic segmentation, change detection and several domain generalization settings, where it obtains strong overall results.
Critical strengths
-
I find the motivation quite convincing: the importance of one spectral band should depend not only on its own identity/content, but also on the other bands available in the current observation. The configuration-aware fusion used in SADBT is therefore a reasonable design, and this is the part of the paper I find most interesting.I don't think anyone else in AI4EO has realized this before.
-
The paper evaluates classification, segmentation, change detection and domain generalization across different sensing settings so i think it's a pretty comprehensive evaluation. The improvements on semantic segmentation are particularly substantial, which supports the claim that the learned patch representations are useful for dense downstream tasks.
Critical weaknesses
-
The author cited DEO from cvpr 2026. DEO is already a very recent dual-teacher EO framework, and my impression is that there is considerable similarity in the overall training philosophy. The current related work section mentions DEO, but does not make the method difference explicit. This makes it somewhat difficult for me to judge how much of the contribution comes from a new training framework versus a new spectral tokenizer...
-
The compute side of this model/method is not transparent, so we have no idea what would be the cost to reach SOTA over other models.
Questions for authors
Some major concerns:
- Could the authors explain more explicitly what are the main differences from DEO, especially for the dual-teacher part? preferably adding this in the main text.
- Could the authors explain why DoY information is not included, given that it is used in quite a few other RSFMs? Is the intention to treat seasonal/time variation mainly as a nuisance factor and learn invariance to it?
- Could the authors please report the model params and forward FLOPs, preferably together with comparable numbers for the main baselines? At the moment it is difficult to judge the computational cost of the reported SOTA results...
- In Fig 2, why is spatial masking used for the global views but not for the local views? Is there an ablation or other evidence supporting this choice?
- Have the authors checked whether there is geographical overlap between the large-scale pretraining data and any of the downstream benchmark datasets? If yes, could the authors clarify how this was handled?
As a minor suggestion: I think AEF and Tessera are also relevant to the multimodal EO foundation model discussion, and could be mentioned in the related work. Their feature visualizations may also provide useful context for Fig 4 / Fig A1.
If the authors can clarify these points, I would consider raising my score.
4 comments
Follow-up questions
I would thank all the authors for the detailed response and the revised pdf. Most of them are clearer right now, but I still have a few questions:
- My question on DEO was mainly about where the gain comes from. The new DINOv3 row in Table 1 makes this more interesting. DINOv3 ViT-B already gets 73.40 miou on segmentation, and if m-Cashew (only 50 test images) is left out, it is basically tied with ScalSpectra. since ScalSpectra is distilled from DINOv3 ViT-L, how much of the dense gain do the authors think comes from the visual prior rather than from MLSRL/CMSRD? also DINOv3 is only in table 1, while change detection and DG are mostly HR RGB, where i’d expect DINOv3 to be quite strong. would the “leads on all benchmarks” claim still hold true there?
- Table A3 is helpful but it’s only at 4 bands. SADBT processes each band separately, so i wonder how the cost looks at 13, 32 (WHU-OHS) or 202 bands, and at what point the tokenizer costs more than the backbone... and how many GPU hours does a full pretraining run take, including the ViT-L teacher? A back of the envelope estimate in the comment would also be fine if authors didn’t keep track of their training processes.
Dense Gains, DINOv3 Comparison, and Spectral Scaling
Thank you for the follow-up questions. We clarify dense-gain sources, compare against DINOv3 Web-L, and report spectral scaling and pretraining cost.
1. Dense gains and DINOv3. The DINOv3 visual prior supplies semantic supervision; CMSRD transfers it from aligned RGB teacher views to multimodal student features. MLSRL adds structural supervision from the EMA multimodal teacher. With SADBT fixed, Table 5 reports:
| Dense-learning configuration | Avg. mIoU | Gain |
|---|---|---|
| SADBT + scene learning and masked-patch iBOT | 70.18 | 0.00 |
| + MLSRL | 73.17 | +2.99 |
| + CMSRD | 74.11 | +3.93 |
| + MLSRL and CMSRD | 75.41 | +5.23 |
Gains are relative to SADBT. MLSRL extends masked-patch iBOT to all patches, improving mIoU by 2.99 without frozen ViT-L distillation during EO pretraining. CMSRD adds 3.93 through teacher-guided transfer. Together, they exceed CMSRD alone by 1.30 and MLSRL alone by 2.24 points. CMSRD's gain reflects the teacher and transfer objective jointly. Under the shared recipe, four tokenizer architectures span 0.66 mIoU (Appendix F).
The earlier 73.40 mIoU used Web ViT-B to match the student size. Tables 1–4 now use DINOv3 Web ViT-L/16 (LVD-1689M), the same checkpoint as our frozen visual-prior teacher; ScalSpectra remains ViT-B. GeoBench segmentation averages are 75.41 vs. 74.13. The 50-image m-Cashew task has equal weight; excluding it gives means of 75.26 for ScalSpectra and 77.08 for DINOv3. DINOv3 leads on RGB m-PV4Ger and m-NeonTree; ScalSpectra leads on RGBN m-Chesapeake (73.61 vs. 73.21) and 12-band m-Cashew (75.84 vs. 65.28).
We added DINOv3 to high-resolution change detection and domain generalization. Encoders are frozen; heads, resolutions, and schedules are matched per benchmark, and DG heads train on source domains.
| Task-group average | DINOv3 Web-L | DEO | ScalSpectra |
|---|---|---|---|
| Change detection, F1 ↑ | 76.51 | 76.27 | 77.02 |
| DG segmentation, mIoU/mAcc ↑ | 54.18/67.98 | 53.96/68.38 | 55.47/69.70 |
| DG height, MAE/RMSE (m) ↓ | 1.756/3.249 | 1.759/3.254 | 1.730/3.163 |
DINOv3 leads on CDD, LEVIR-CD+, LoveDA mIoU, FLAIR mAcc, and GAMUS-to-DFC2019. ScalSpectra exceeds DEO across the reported change-detection and DG settings. Accordingly, our claim is best average performance within each task group, not on every dataset.
On WHU-OHS, EnMAP-NLCD, and Sen1Floods11, mIoU is 56.60/36.62/77.02 for ScalSpectra, 46.82/34.75/75.62 for DINOv3, and 40.08/30.31/75.06 for DEO. ScalSpectra uses 32/202 hyperspectral bands or VV/VH; DINOv3 uses RGB subsets or [VV, VH, 0]. Their task-group averages are 56.75/52.40/48.48. Thus, ScalSpectra transfers better on the evaluated multispectral and hyperspectral tasks and remains comparable to DINOv3 on RGB remote-sensing tasks.
2. Spectral scaling and pretraining budget. We profiled 11 baselines and ScalSpectra at native image sizes, retaining native patch sizes and specified band mappings. Counts are batch-one encoder forwards with 1 MAC = 2 FLOPs, excluding decoders, normalization, nonlinearities, elementwise operations, and preprocessing. SADBT has 5.509M parameters and ViT-B/16 85.079M, totaling 90.588M regardless of band count.
| Input | SADBT GFLOPs | ViT-B GFLOPs | Total GFLOPs | Crossover bands |
|---|---|---|---|---|
| S2, 13 bands, 64×64 | 1.847 | 3.584 | 5.430 | 26 |
| WHU-OHS, 32 bands, 512×512 | 289.306 | 213.829 | 503.135 | 24 |
| EnMAP, 202 bands, 128×128 | 120.090 | 11.896 | 131.986 | 22 |
At fixed spatial size, SADBT first exceeds ViT-B in computational cost at 26 bands for 64×64 inputs, 24 for 512×512, and 22 for 128×128. For example, at 512×512, SADBT costs 207.967 GFLOPs at 23 bands and 216.980 at 24, compared with 213.829 for ViT-B. Tokenization can therefore dominate with full-spectrum hyperspectral inputs. EnMAP's lower absolute cost than WHU-OHS reflects its smaller spatial size. At fixed resolution, ViT-B cost is independent of band count because SADBT produces one fused token per spatial patch.
The full 20-epoch pretraining run took 6 days, 22:41:42 on eight A100 80-GB GPUs, approximately 1,333.56 GPU-hours.
I want to thank the authors for the detailed response. It's a nice write-up and my concerns have been resolved, so I've decided to raise my score to 3.
ScalSpectra's advances over DEO, dual-teacher supervision, and computational cost
Thank you for recognizing our configuration-aware fusion and broad evaluation. We clarify our contributions over DEO and report compute and evaluation details.
1. ScalSpectra's differences from DEO and substantive advances. Both use SimDINOv2's scene-level cosine-compression and coding-rate-expansion objective. ScalSpectra advances variable-band tokenization and dense multimodal learning by combining visual semantic transfer with local structural supervision.
DEO uses separate embeddings for 3-channel RGB and fixed 10-band Sentinel-2; its frozen visual teacher matches CLS and patch features on the student's RGB pathway. ScalSpectra's SADBT uses a shared tokenizer and a spectral-adaptive query derived from observed descriptors for variable band sets. In CMSRD, the teacher receives aligned RGB bands while the student processes corresponding RGB, multispectral, or hyperspectral views. Student features serve as queries and values, teacher patches as keys; outputs align with spatially corresponding teacher patches, transferring visual semantics to multimodal features. MLSRL adds full-grid alignment of masked and visible patches with an unmasked EMA multimodal teacher (Sections 3.3–3.4).
Table 5 separates component benefits within ScalSpectra. Replacing the Panopticon-style embedding with SADBT raises linear probing 70.83 → 71.73 and mIoU 70.04 → 70.18. With SADBT fixed, MLSRL and CMSRD yield 73.17/74.11 mIoU, and together 75.41. Complete-model comparisons against DEO give:
These results appear in Tables 1–4; Figures 4–6 compare frozen multispectral, hyperspectral, and RGB features.
2. Computational cost. Appendix C.4/Table A3 profiles all 11 baselines and ScalSpectra on a common 512 × 512 four-band input (batch 1), retaining native patch sizes and band mappings. Selected results:
We count active encoder multiply-adds with 1 MAC = 2 FLOPs. ScalSpectra includes SADBT and one exported EMA ViT-B, excluding the pretraining student, visual teacher, and training heads. At 512 × 512, DINOv3, DOFA, and ScalSpectra use 16-pixel patches (1,024 spatial tokens), DEO 8-pixel patches (4,096), and Panopticon stride 14 (1,296). Attention matrix products scale as N2: DEO's 4× token count yields ~16× the attention-matrix FLOPs of a 1,024-token model; Panopticon's 1,296 tokens yield ~1.6×. Embeddings also contribute. EO pretraining uses 20 epochs on eight 80-GB A100 GPUs.
3. DoY and temporal sampling. We omit day-of-year (DoY) metadata, treating seasonal and acquisition-time variation primarily as nuisance factors. When revisits exist, independent time sampling per physical band creates temporally varied views of the same location. Scene-level self-distillation encourages stable semantics without synchronized acquisitions (Appendix C.1).
4. Global-view masking. Following DINOv2 and DINOv3, we mask global crops and leave local crops unmasked. Global crops support MLSRL's masked-patch recovery and visible-patch anchoring. In DINO-style self-supervision, local views help learn local features; keeping them unmasked preserves the needed image content. Table 5 shows MLSRL raises mIoU 70.18 → 73.17.
5. Geographic overlap. Pretraining uses four public datasets: OpticalRS-4M (RGB), EarthView (Satellogic, Sentinel-1/2), SpectralEarth (EnMAP hyperspectral), and MMEarth (co-located modalities). Following the original EarthView and MMEarth studies, we evaluate these pretrained representations on GeoBench. Additional change-detection, segmentation, and height datasets use different sensors, testing transfer across tasks and acquisition conditions. Frozen encoders support task decoders, and domain-generalization decoders train only on the source domain before direct target evaluation. Shared locations would still allow cross-sensor transfer evaluation. We have not independently audited geographic overlap.
6. Related work. Section 2 cites AlphaEarth Foundations and TESSERA and discusses global embedding fields and pixel-wise time series.