acceptodds
Under review as a conference paper at ICLR 2027

ScalSpectra: A Scalable Any-Spectral Foundation Model for Earth Observation

40 views

Abstract

We propose ScalSpectra, a self-supervised framework for multimodal Earth observation (EO) foundation model pretraining. Although several pioneering efforts have emerged in this field, existing approaches still suffer from two major limitations. First, different EO sensors provide different sets of spectral bands, and the contribution of each band to the fused representation depends on which other bands are observed alongside it. Current tokenizers do not explicitly adapt band fusion to the observed band set, leaving this cross-band dependency unmodeled in the fused representation. Second, existing multimodal EO foundation models predominantly rely on masked image modeling and contrastive self-distillation objectives that require large-scale, high-quality data to learn dense semantic features, whereas spectral observations are relatively scarce, costly to acquire, and complex to process. Our framework is built on spectral-aware dynamic band tokenization (SADBT) and dual-teacher dense feature learning. SADBT encodes band content together with optical wavelengths or SAR polarization and orbit descriptors, and uses a configuration-aware query to fuse the available bands into physically informed patch embeddings, so a single model scales to any band combination. The dual-teacher framework combines multimodal local structural representation learning (MLSRL), which aligns masked student features with an unmasked exponential moving average teacher over all patch positions, and cross-modal semantic representation distillation (CMSRD), which transfers dense semantic knowledge from a frozen visual-prior teacher through aligned optical observations while retaining modality-specific evidence. We pretrain ScalSpectra on heterogeneous EO data sources spanning RGB, multispectral, hyperspectral, and SAR observations. Extensive evaluations across scene classification, semantic segmentation, change detection, and domain- generalized dense prediction demonstrate that ScalSpectra achieves the best overall performance across the evaluated benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Earth Observation, Remote Sensing Foundation Model, Cross-Modal Distillation, Multispectral

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_i2Y9OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a self-supervised pretraining framework for multimodal Earth observation foundation models called ScalSpectra. The goal is to build a unified model that can work with different spectral configurations, including RGB, multispectral and SAR, etc. observations, while also learning good dense patch representations.

A main component is SADBT, which uses physical band descriptors and a configuration-aware query to fuse the available bands. The important idea is that the contribution of a band depends not only on the band itself, but also on which other bands are observed together with it. The framework further uses dual-teacher dense feature learning to improve the local and semantic quality of the patch features. The model is evaluated on scene classification, semantic segmentation, change detection and several domain generalization settings, where it obtains strong overall results.

Critical strengths
  1. I find the motivation quite convincing: the importance of one spectral band should depend not only on its own identity/content, but also on the other bands available in the current observation. The configuration-aware fusion used in SADBT is therefore a reasonable design, and this is the part of the paper I find most interesting.I don't think anyone else in AI4EO has realized this before.

  2. The paper evaluates classification, segmentation, change detection and domain generalization across different sensing settings so i think it's a pretty comprehensive evaluation. The improvements on semantic segmentation are particularly substantial, which supports the claim that the learned patch representations are useful for dense downstream tasks.

Critical weaknesses
  1. The author cited DEO from cvpr 2026. DEO is already a very recent dual-teacher EO framework, and my impression is that there is considerable similarity in the overall training philosophy. The current related work section mentions DEO, but does not make the method difference explicit. This makes it somewhat difficult for me to judge how much of the contribution comes from a new training framework versus a new spectral tokenizer...

  2. The compute side of this model/method is not transparent, so we have no idea what would be the cost to reach SOTA over other models.

Questions for authors

Some major concerns:

  1. Could the authors explain more explicitly what are the main differences from DEO, especially for the dual-teacher part? preferably adding this in the main text.
  2. Could the authors explain why DoY information is not included, given that it is used in quite a few other RSFMs? Is the intention to treat seasonal/time variation mainly as a nuisance factor and learn invariance to it?
  3. Could the authors please report the model params and forward FLOPs, preferably together with comparable numbers for the main baselines? At the moment it is difficult to judge the computational cost of the reported SOTA results...
  4. In Fig 2, why is spatial masking used for the global views but not for the local views? Is there an ablation or other evidence supporting this choice?
  5. Have the authors checked whether there is geographical overlap between the large-scale pretraining data and any of the downstream benchmark datasets? If yes, could the authors clarify how this was handled?

As a minor suggestion: I think AEF and Tessera are also relevant to the multimodal EO foundation model discussion, and could be mentioned in the related work. Their feature visualizations may also provide useful context for Fig 4 / Fig A1.

If the authors can clarify these points, I would consider raising my score.

4 comments
AuthorsOpenReview ↗

ScalSpectra's advances over DEO, dual-teacher supervision, and computational cost

Thank you for recognizing our configuration-aware fusion and broad evaluation. We clarify our contributions over DEO and report compute and evaluation details.

1. ScalSpectra's differences from DEO and substantive advances. Both use SimDINOv2's scene-level cosine-compression and coding-rate-expansion objective. ScalSpectra advances variable-band tokenization and dense multimodal learning by combining visual semantic transfer with local structural supervision.

DEO uses separate embeddings for 3-channel RGB and fixed 10-band Sentinel-2; its frozen visual teacher matches CLS and patch features on the student's RGB pathway. ScalSpectra's SADBT uses a shared tokenizer and a spectral-adaptive query derived from observed descriptors for variable band sets. In CMSRD, the teacher receives aligned RGB bands while the student processes corresponding RGB, multispectral, or hyperspectral views. Student features serve as queries and values, teacher patches as keys; outputs align with spatially corresponding teacher patches, transferring visual semantics to multimodal features. MLSRL adds full-grid alignment of masked and visible patches with an unmasked EMA multimodal teacher (Sections 3.3–3.4).

Table 5 separates component benefits within ScalSpectra. Replacing the Panopticon-style embedding with SADBT raises linear probing 70.83 → 71.73 and mIoU 70.04 → 70.18. With SADBT fixed, MLSRL and CMSRD yield 73.17/74.11 mIoU, and together 75.41. Complete-model comparisons against DEO give:

EvaluationDEOScalSpectra
GeoBench avg. segmentation mIoU68.0975.41
Change detection avg. binary F176.2777.02
WHU-OHS hyperspectral mIoU40.0856.60
Domain-generalization avg. segmentation mIoU53.9655.47
Height estimation avg. MAE/RMSE (m)1.759/3.2541.730/3.163

These results appear in Tables 1–4; Figures 4–6 compare frozen multispectral, hyperspectral, and RGB features.

2. Computational cost. Appendix C.4/Table A3 profiles all 11 baselines and ScalSpectra on a common 512 × 512 four-band input (batch 1), retaining native patch sizes and band mappings. Selected results:

EncoderDINOv3DOFAPanopticonDEOScalSpectra
Params (M)85.670111.31198.97686.30090.588
Forward GFLOPs215.037214.825401.7251315.940251.754

We count active encoder multiply-adds with 1 MAC = 2 FLOPs. ScalSpectra includes SADBT and one exported EMA ViT-B, excluding the pretraining student, visual teacher, and training heads. At 512 × 512, DINOv3, DOFA, and ScalSpectra use 16-pixel patches (1,024 spatial tokens), DEO 8-pixel patches (4,096), and Panopticon stride 14 (1,296). Attention matrix products scale as N2N^2: DEO's 4× token count yields ~16× the attention-matrix FLOPs of a 1,024-token model; Panopticon's 1,296 tokens yield ~1.6×. Embeddings also contribute. EO pretraining uses 20 epochs on eight 80-GB A100 GPUs.

3. DoY and temporal sampling. We omit day-of-year (DoY) metadata, treating seasonal and acquisition-time variation primarily as nuisance factors. When revisits exist, independent time sampling per physical band creates temporally varied views of the same location. Scene-level self-distillation encourages stable semantics without synchronized acquisitions (Appendix C.1).

4. Global-view masking. Following DINOv2 and DINOv3, we mask global crops and leave local crops unmasked. Global crops support MLSRL's masked-patch recovery and visible-patch anchoring. In DINO-style self-supervision, local views help learn local features; keeping them unmasked preserves the needed image content. Table 5 shows MLSRL raises mIoU 70.18 → 73.17.

5. Geographic overlap. Pretraining uses four public datasets: OpticalRS-4M (RGB), EarthView (Satellogic, Sentinel-1/2), SpectralEarth (EnMAP hyperspectral), and MMEarth (co-located modalities). Following the original EarthView and MMEarth studies, we evaluate these pretrained representations on GeoBench. Additional change-detection, segmentation, and height datasets use different sensors, testing transfer across tasks and acquisition conditions. Frozen encoders support task decoders, and domain-generalization decoders train only on the source domain before direct target evaluation. Shared locations would still allow cross-sensor transfer evaluation. We have not independently audited geographic overlap.

6. Related work. Section 2 cites AlphaEarth Foundations and TESSERA and discusses global embedding fields and pixel-wise time series.

Reviewer_i2Y9OpenReview ↗

Follow-up questions

I would thank all the authors for the detailed response and the revised pdf. Most of them are clearer right now, but I still have a few questions:

  1. My question on DEO was mainly about where the gain comes from. The new DINOv3 row in Table 1 makes this more interesting. DINOv3 ViT-B already gets 73.40 miou on segmentation, and if m-Cashew (only 50 test images) is left out, it is basically tied with ScalSpectra. since ScalSpectra is distilled from DINOv3 ViT-L, how much of the dense gain do the authors think comes from the visual prior rather than from MLSRL/CMSRD? also DINOv3 is only in table 1, while change detection and DG are mostly HR RGB, where i’d expect DINOv3 to be quite strong. would the “leads on all benchmarks” claim still hold true there?
  2. Table A3 is helpful but it’s only at 4 bands. SADBT processes each band separately, so i wonder how the cost looks at 13, 32 (WHU-OHS) or 202 bands, and at what point the tokenizer costs more than the backbone... and how many GPU hours does a full pretraining run take, including the ViT-L teacher? A back of the envelope estimate in the comment would also be fine if authors didn’t keep track of their training processes.
AuthorsOpenReview ↗

Dense Gains, DINOv3 Comparison, and Spectral Scaling

Thank you for the follow-up questions. We clarify dense-gain sources, compare against DINOv3 Web-L, and report spectral scaling and pretraining cost.

1. Dense gains and DINOv3. The DINOv3 visual prior supplies semantic supervision; CMSRD transfers it from aligned RGB teacher views to multimodal student features. MLSRL adds structural supervision from the EMA multimodal teacher. With SADBT fixed, Table 5 reports:

Dense-learning configurationAvg. mIoUGain
SADBT + scene learning and masked-patch iBOT70.180.00
+ MLSRL73.17+2.99
+ CMSRD74.11+3.93
+ MLSRL and CMSRD75.41+5.23

Gains are relative to SADBT. MLSRL extends masked-patch iBOT to all patches, improving mIoU by 2.99 without frozen ViT-L distillation during EO pretraining. CMSRD adds 3.93 through teacher-guided transfer. Together, they exceed CMSRD alone by 1.30 and MLSRL alone by 2.24 points. CMSRD's gain reflects the teacher and transfer objective jointly. Under the shared recipe, four tokenizer architectures span 0.66 mIoU (Appendix F).

The earlier 73.40 mIoU used Web ViT-B to match the student size. Tables 1–4 now use DINOv3 Web ViT-L/16 (LVD-1689M), the same checkpoint as our frozen visual-prior teacher; ScalSpectra remains ViT-B. GeoBench segmentation averages are 75.41 vs. 74.13. The 50-image m-Cashew task has equal weight; excluding it gives means of 75.26 for ScalSpectra and 77.08 for DINOv3. DINOv3 leads on RGB m-PV4Ger and m-NeonTree; ScalSpectra leads on RGBN m-Chesapeake (73.61 vs. 73.21) and 12-band m-Cashew (75.84 vs. 65.28).

We added DINOv3 to high-resolution change detection and domain generalization. Encoders are frozen; heads, resolutions, and schedules are matched per benchmark, and DG heads train on source domains.

Task-group averageDINOv3 Web-LDEOScalSpectra
Change detection, F1 ↑76.5176.2777.02
DG segmentation, mIoU/mAcc ↑54.18/67.9853.96/68.3855.47/69.70
DG height, MAE/RMSE (m) ↓1.756/3.2491.759/3.2541.730/3.163

DINOv3 leads on CDD, LEVIR-CD+, LoveDA mIoU, FLAIR mAcc, and GAMUS-to-DFC2019. ScalSpectra exceeds DEO across the reported change-detection and DG settings. Accordingly, our claim is best average performance within each task group, not on every dataset.

On WHU-OHS, EnMAP-NLCD, and Sen1Floods11, mIoU is 56.60/36.62/77.02 for ScalSpectra, 46.82/34.75/75.62 for DINOv3, and 40.08/30.31/75.06 for DEO. ScalSpectra uses 32/202 hyperspectral bands or VV/VH; DINOv3 uses RGB subsets or [VV, VH, 0]. Their task-group averages are 56.75/52.40/48.48. Thus, ScalSpectra transfers better on the evaluated multispectral and hyperspectral tasks and remains comparable to DINOv3 on RGB remote-sensing tasks.

2. Spectral scaling and pretraining budget. We profiled 11 baselines and ScalSpectra at native image sizes, retaining native patch sizes and specified band mappings. Counts are batch-one encoder forwards with 1 MAC = 2 FLOPs, excluding decoders, normalization, nonlinearities, elementwise operations, and preprocessing. SADBT has 5.509M parameters and ViT-B/16 85.079M, totaling 90.588M regardless of band count.

InputSADBT GFLOPsViT-B GFLOPsTotal GFLOPsCrossover bands
S2, 13 bands, 64×641.8473.5845.43026
WHU-OHS, 32 bands, 512×512289.306213.829503.13524
EnMAP, 202 bands, 128×128120.09011.896131.98622

At fixed spatial size, SADBT first exceeds ViT-B in computational cost at 26 bands for 64×64 inputs, 24 for 512×512, and 22 for 128×128. For example, at 512×512, SADBT costs 207.967 GFLOPs at 23 bands and 216.980 at 24, compared with 213.829 for ViT-B. Tokenization can therefore dominate with full-spectrum hyperspectral inputs. EnMAP's lower absolute cost than WHU-OHS reflects its smaller spatial size. At fixed resolution, ViT-B cost is independent of band count because SADBT produces one fused token per spatial patch.

The full 20-epoch pretraining run took 6 days, 22:41:42 on eight A100 80-GB GPUs, approximately 1,333.56 GPU-hours.

I want to thank the authors for the detailed response. It's a nice write-up and my concerns have been resolved, so I've decided to raise my score to 3.

Reviewer_y9tHOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work trains a foundation model for Earth Observation that aims to encode any spectral observation. It uses SimDINOv2 training with DINOv3 distillation.

Critical strengths

Beyond GeoBench, the paper covers change detection, source-only domain generalization for segmentation, and height estimation, including a synthetic-to-real shift.

Critical weaknesses

The main weakness is that the paper lacks some really important comparisons/related work. First, papers that need to be evaluated against and/or credited for parts of the contributions:

  • DINOv3 is missing as a baseline, which is essential since it is the distillation teacher.
  • Panopticon is the closest tokenizer, but it is absent from the main tables, and its SAR embedding is reused without credit.
  • DOFA is compared, but its band-set-dependent hypernetwork is not discussed. DOFA++ has been out for a moment as well and could be compared to.
  • SimDINO / SimDINOv2 should be credited as the baseline recipe.
  • OlmoEarth (CVPR 2026) and TerraFM (ICLR 2026) need to be compared as they are much more recent state of the art. Then the paper lacks important contributions of the field in the related work: previously cited papers + RAMEN (CVPR 2026) and TerraMind (ICCV 2025), at minimum.

The “any-spectral” claim is untested: most datasets are all RGB or RGBN or even S2, which is where DINOv3 distillation helps most. There is no hyperspectral or SAR-only downstream task.

The claim that prior tokenizers ignore the observed band set is inaccurate. DOFA’s Transformer weight generator attends jointly over all input wavelengths. Panopticon’s softmax pooling also depends implicitly on which bands are present.

Questions for authors

How does the DINOv3 model perform on the benchmarks and which one is it?

Can you ablate each SADBT component and compare SADBT with DOFA-, Panopticon- and RAMEN- style tokenizers under the same backbone and objective?

How does the model perform on a hyperspectral task and a SAR-only task?

4 comments
AuthorsOpenReview ↗

Expanded Baselines, Spectral-Adaptive Tokenization, and Cross-Spectral Evaluation

1. Expanded comparisons, DINOv3 identity, and attribution. Table 1 now includes DINOv3, Panopticon, OlmoEarth, and TerraFM. The DINOv3 baseline is the official Web ViT-B/16 (LVD-1689M) model, using dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth. Its ViT-Base backbone matches ScalSpectra's, controlling for model capacity in a fair comparison. Results averaged over four GeoBench classification and four segmentation datasets are:

MethodAvg. classificationAvg. segmentation mIoU
DINOv3 Web ViT-B/16 (2025)69.7373.40
Panopticon (2025)70.3166.62
OlmoEarth (2026)66.6762.79
TerraFM (2026)70.2965.26
ScalSpectra (ours)71.7675.41

ScalSpectra exceeds the strongest classification baseline, Panopticon, by 1.45 points, and the strongest segmentation baseline, DINOv3, by 2.01 points. Segmentation freezes the encoder and trains only the decoder, providing quantitative evidence of pretrained dense-feature transfer.

We have also clarified the sources of inherited components: Section 3.3 credits Panopticon's SAR descriptor encoding, including transmit/receive polarization and orbit direction; Section 3.4 credits SimDINOv2's cosine compression and coding-rate expansion as the scene-level baseline. Section 2 now discusses DOFA's band-set-dependent hypernetwork and cites OlmoEarth, TerraFM, RAMEN, and TerraMind. The DOFA baseline uses the latest DOFAv2 weights released through the official DOFA GitHub repository.

2. Hyperspectral and SAR-only validation. We added WHU-OHS, using all 32 hyperspectral bands, and SARBuD, using SAR imagery only. Both evaluations freeze the pretrained backbone and train a Mask2Former decoder. Table 3 reports mIoU (%) as follows:

MethodWHU-OHSSARBuDAverage
CROMA31.7072.3152.01
SenPaMAE42.8871.6457.26
Prithvi-v243.4472.7258.08
DOFA40.2272.2756.25
Galileo31.2572.6851.97
Copernicus-FM31.0272.3151.67
DEO40.0872.5456.31
ScalSpectra56.6073.0164.81

ScalSpectra improves over the strongest baseline, Prithvi-v2, by 13.16 points on WHU-OHS and 0.29 points on SARBuD. These tasks extend the evidence beyond RGB and multispectral observations to hyperspectral and SAR-only inputs. Dataset descriptions and evaluation protocols appear in Sections 4.1–4.2.

3. Observed-band dependence, SADBT differences, and controlled ablations. The revised Abstract and Introduction clarify that DOFA and Panopticon both use the observed band set. DOFA's Transformer jointly processes all input wavelengths to generate band-projection weights. These weights are shared across spatial locations and generated without band image content. Panopticon adds wavelength encodings to independently projected band features and uses them as keys and values for a shared learned query. Its softmax weights depend on the available bands, but the query itself is not generated from their wavelengths, and band features do not interact before pooling.

SADBT first uses physical descriptors as queries to attend to all available band features at each patch, allowing them to incorporate cross-band content before fusion. A shared learnable seed then summarizes the observed descriptors into a spectral-adaptive query. This query varies with the band combination and guides aggregation at each patch; the fusion weights also depend on that patch's band features. Thus, the physical composition of the current observation directly guides fusion.

Table 5 isolates SADBT's effect against the Panopticon-style band embedding, the closest tokenizer to our design. We keep the ViT-B backbone and SimDINOv2 objective fixed, including the baseline iBOT term, and replace only the tokenizer:

TokenizerAvg. linear probingAvg. segmentation mIoU
Panopticon-style embedding70.8370.04
SADBT71.7370.18

The +0.90 linear-probing gain, compared with +0.14 mIoU, suggests that this replacement primarily strengthens global, scene-level representations while also modestly improving dense prediction.

I thank the authors for the detailed rebuttal. However, several soundness concerns remain:

  • The paper’s distillation teacher is DINOv3 ViT-L, but the rebuttal reports DINOv3 ViT-B. Please report the actual teacher (and say whether it is the Web or Sat variant), and add DINOv3 to the change-detection, domain-generalization and height-estimation tables. These are all RGB/RGBN, which is where the visual prior should be strongest.
  • About the new evaluations, these datasets are not mainstream ones for evaluating GeoFM on such tasks. The most used ones are SpectralEarthBench and Sen1Floods. 32 bands are not really characteristic of hyperspectral data either.
  • About the ablation, the new experiment comparison has the same numbers as rows 1–2 of Table 4 in the submission. Specifically, SADBT's three parts have still not been ablated. The novelty claim rests on the configuration-aware query, but no experiment isolates it, so the gain may come from cross-band interaction alone. The SADBT effect is never measured in the full model: Table 4 has no row with the Panopticon-style embedding plus MLSRL and CMSRD, so it is unclear whether SADBT contributes anything once DINOv3 distillation is added. The ablation uses GeoBench tasks with fixed, mostly RGB/RGBN band sets, which is where tokenizer design matters least. To support the any-spectral claim, please run the tokenizer comparison (including DOFA- and RAMEN-style tokenizers) on more diverse datasets.
AuthorsOpenReview ↗

Actual-Teacher Comparison, Expanded Spectral Benchmarks, and Tokenizer Evidence

Thank you for the follow-up comments. We have clarified the teacher identity, expanded the spectral benchmarks, and added matched tokenizer comparisons.

1. Comparison with the actual teacher. Our frozen visual-prior teacher is DINOv3 Web ViT-L/16 (LVD-1689M), using dinov3_vitl16_pretrain_lvd1689m-8aa4cbdd.pth. The earlier Web ViT-B baseline matched ScalSpectra's backbone size. We now use the same Web ViT-L checkpoint as our visual-prior teacher for all downstream comparisons, including change detection, domain-generalized segmentation, and height estimation (Tables 1–4).

Task groupMetricDINOv3 Web-LScalSpectra
GeoBench classificationAverage ↑69.8971.76
GeoBench segmentationmIoU ↑74.1375.41
Change detectionF1 ↑76.5177.02
Domain-generalized segmentationmIoU/mAcc ↑54.18/67.9855.47/69.70
Height estimationMAE/RMSE (m) ↓1.756/3.2491.730/3.163

All entries are task-group averages. The models are competitive on high-resolution RGB change detection and height estimation, with DINOv3 leading on some individual tasks. ScalSpectra shows a larger gain on multispectral m-Cashew-Plant, achieving 75.84 versus 65.28 mIoU (+10.56 points). The revised conclusion concerns the highest average performance within each evaluated task group.

2. Expanded hyperspectral and SAR evaluation. We added 202-band EnMAP-NLCD from SpectralEarth and Sen1Floods11 with Sentinel-1 VV/VH, complementing 32-band WHU-OHS. Each uses official training, validation, and test splits, a frozen pretrained backbone, and matched Mask2Former decoder settings (Sections 4.1–4.2 and Appendix B.2).

DatasetScalSpectra mIoUStrongest baselineGain
WHU-OHS, 32 bands56.60DINOv3, 46.82+9.78
EnMAP-NLCD, 202 bands36.62Galileo, 35.20+1.42
Sen1Floods11, VV/VH77.02Galileo, 76.58+0.44
Three-task average56.75DINOv3, 52.40+4.35

ScalSpectra uses all 32 WHU-OHS bands and all 202 retained EnMAP bands; DINOv3 uses their RGB subsets, obtaining 46.82 and 34.75 mIoU. These results demonstrate effective frozen-feature transfer with full-spectrum hyperspectral inputs and SAR flood imagery.

3. Tokenizer configurations and learning objectives. The Panopticon/SADBT comparison corresponds to submission Table 4, rows 1–2, now Table 5. Section 4.4 explicitly identifies row 1 as ViT-B with the complete Panopticon tokenizer, the SimDINOv2 scene-level objective, and masked-patch iBOT. Row 2 replaces the Panopticon tokenizer with SADBT, holding the backbone, data, and 20-epoch recipe fixed. Linear probing/segmentation improves from 70.83/70.04 to 71.13/70.18 (+0.30/+0.14 points). This evaluates complete tokenizer architectures, including their cross-band interaction and query-generation mechanisms.

Appendix F, Table A4 adds matched DOFA-style and RAMEN-style comparisons using the same backbone, data, schedule, objectives, and downstream protocol.

TokenizerLinear probingSegmentationOverall
Panopticon-style70.8370.0470.44
DOFA-style70.8769.5270.20
RAMEN-style70.0869.7069.89
SADBT71.1370.1870.66

Entries are task-group averages; Overall averages the displayed classification and segmentation scores. GeoBench spans 3–13 channels. Classification uses 13/6/12/10 bands for m-EuroSAT/m-ForestNet/m-BigEarthNet/m-So2Sat; segmentation uses RGB for m-PV4Ger-Seg and m-NeonTree, RGBN for m-Chesapeake, and 12 bands for m-Cashew-Plant. Five tasks use at least six optical bands. Within each task, tokenizers receive identical bands, splits, and evaluation heads.

The classification and segmentation spreads across tokenizers are 1.05 and 0.66 points. These architectures enable flexible spectral inputs, with modest average-performance differences under the shared recipe. With SADBT fixed, MLSRL and CMSRD improve segmentation by 2.99 and 3.93 points; together they improve it by 5.23 points, from 70.18 to 75.41, while classification rises from 71.13 to 71.76. These results show larger dense-transfer gains from the training objectives in the tested configurations. Flexible spectral embedding and dense semantic learning thus serve complementary roles in ScalSpectra.

I thank the authors for the additional experiments. They answer my questions about the teacher’s identity and the tokenizer baselines. However, they reinforce rather than resolve my main concerns.

First, on the evaluations against the teacher, the whole average gain comes from m-Cashew-Plant, whose test split contains only 50 patches.

By the authors’ own numbers, the four tokenizers are within 1.05 / 0.66 points, and SADBT improves on the Panopticon-style tokenizer by only +0.30 / +0.14. The component-wise SADBT ablation, the Panopticon-style embedding with MLSRL and CMSRD, and a tokenizer comparison on the hyperspectral and SAR tasks are still missing.

For the new evaluations on hyperspectral, use models suited for hyperspectral, such as Panopticon or DOFA. Galileo is not a strong or relevant competitor. Moreover, report micro IoU, as SpectralEarth to have comparison with their numbers.

On the evaluation on Sen1Flood, OlmoEarth reports 78–80 mIoU for all models it tested, using linear probing, while ScalSpectra reaches 77.02 with a Mask2Former decoder.

The central claim, a scalable any-spectral foundation model enabled by SADBT, remains insufficiently supported. I maintain my rating.

Reviewer_my7tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ScalSpectra combines spectral-aware dynamic band tokenization (SADBT) with dual-teacher dense feature learning for multimodal Earth observation. SADBT incorporates optical wavelengths or SAR polarization and orbit descriptors to aggregate variable band sets into fixed-dimensional patch embeddings. MLSRL supervises all patch positions using an EMA teacher, while CMSRD transfers dense semantics from a frozen visual-prior teacher through aligned optical observations. Evaluations cover classification, segmentation, change detection, and domain generalization, with the highest average GeoBench classification and segmentation scores among the evaluated models.

Critical strengths
  1. ScalSpectra combines shared per-band patch projection with descriptors encoding optical center wavelengths or SAR polarization and orbit attributes. Mapping different band configurations to fixed-dimensional spatial tokens allows the same backbone to accommodate variable inputs, improving model reusability and flexibility.
  2. The GeoBench segmentation results use a frozen backbone with a trained task decoder, providing relatively direct evidence of pretrained feature transferability. The highest average classification linear-probing score further suggests that the improvements in dense prediction coexist with strong scene-level discrimination.
Critical weaknesses
  1. The methodological advance over closely related approaches is insufficiently established. Prior work has investigated flexible band adaptation and knowledge transfer from pretrained teachers, yet the paper does not clearly distinguish the additional capability or insight provided by its designs. Although the experimental results are sound, the technical contribution appears incremental as currently demonstrated.
  2. The evaluation includes several relevant baselines, but its coverage of recent, directly related methods remains limited. This makes it difficult to assess the reported improvements against the current literature, weakening the persuasiveness of both the benchmark results and the dense feature comparisons.
Questions for authors
  1. What is the substantive advance over related methods that model channel attributes or enhance representations through knowledge distillation?
  2. Could the authors provide a more detailed demonstration and explanation of SADBT?
  3. Given the extensive availability of publicly available RGB, multispectral, hyperspectral, and SAR imagery, why is data scarcity emphasized as a key issue? How do the experiments connect limited data availability to the benefits of dual-teacher learning?
  4. Could the pretrained feature visualizations include greater diversity in modalities and geographic regions, given the relative homogeneity of the current examples?
2 comments
AuthorsOpenReview ↗

Spectral-Adaptive Fusion, Dense Semantic Transfer, and Expanded Evaluation

1. Substantive advance. ScalSpectra's core advances are using the observed physical band set to guide patch-wise fusion and transferring dense visual semantics to multimodal features. Together, these couple an observation-adaptive tokenizer with complementary semantic and structural supervision. DOFA jointly processes observed wavelengths to generate projection weights shared across spatial locations. Panopticon adds wavelength encodings to per-band features and pools them with a shared learned query; its softmax weights implicitly depend on which bands are present. SADBT explicitly derives its fusion query from the current physical descriptor set and conditions each patch's band features on the other observed bands before aggregation. Thus, wavelength-set composition determines the patch-wise fusion context, while patch content determines how that context is applied.

The distillation contribution is likewise more specific than using two teachers. CMSRD uses multimodal student features as queries and values and frozen visual-teacher patch features as keys, then aligns the resulting student features with spatially corresponding teacher targets. Aligned RGB teacher views therefore transfer dense semantics to student features from RGB, multispectral, or hyperspectral inputs. DEO applies visual-teacher matching to its RGB student pathway. MLSRL complements CMSRD by aligning masked and visible student patches with an unmasked EMA multimodal teacher. Both objectives train the same tokenizer and backbone.

2. Expanded recent-baseline coverage. Table 1 adds four directly relevant methods. Their average GeoBench results are:

MethodAvg. classificationAvg. segmentation
DINOv3 Web ViT-B/16 (2025)69.7373.40
Panopticon (2025)70.3166.62
OlmoEarth (2026)66.6762.79
TerraFM (2026)70.2965.26
ScalSpectra (ours)71.7675.41

ScalSpectra exceeds the strongest baseline by 1.45 points in average classification (Panopticon) and 2.01 mIoU points in average segmentation (DINOv3). DINOv3 uses Web ViT-B/16 (LVD-1689M), matching our ViT-Base backbone size. Frozen-backbone segmentation tests pretrained dense-feature transfer, complementing Figures 4–6.

3. SADBT mechanism. For a view with CC available bands, a shared patch projection first produces one content feature per band and patch, while physical descriptors encode the bands' wavelengths. At each patch, descriptor embeddings query all available band features as keys and values, producing cross-band-conditioned features while retaining the CC-band axis. A shared learnable seed then attends to the current descriptor set to produce a spectral-adaptive query. This query changes with the observed band combination and is shared across spatial patches in that view. At each patch, it scores the cross-band-conditioned features; softmax-normalized weights form a weighted sum over bands. The reduction is (1×C)(C×d)=1×d(1\times C)(C\times d)=1\times d, yielding one fixed-width token per patch for any positive CC.

4. Dense-semantic motivation and dual-teacher evidence. Our goal is to learn dense features with strong semantic discriminability for multimodal remote-sensing foundation models. Visual foundation models learn dense semantics through self-supervision on massive, curated RGB corpora. Public EO archives are extensive, but acquiring high-quality, high-resolution multispectral/hyperspectral imagery remains costly; storage and processing demands constrain pretraining scale. On our 384 × 384 EarthView Sentinel-2 grid, one million single-date 12-band observations represent about 1.77 TB of uncompressed uint8-equivalent pixels versus 0.44 TB for RGB. CMSRD reuses RGB-teacher semantics in multimodal student features; MLSRL adds EO structural supervision. With data and the 20-epoch schedule fixed, Table 5 reports mIoU of 70.18 for SADBT, 74.11 (+3.93) with CMSRD, 73.17 (+2.99) with MLSRL, and 75.41 (+5.23) with both, exceeding CMSRD alone by 1.30 points. These frozen-encoder results support complementary semantic and structural learning within the available EO data.

5. More diverse feature visualizations. Figures 4–6 show frozen pretrained features on multispectral Five-Billion-Pixels, hyperspectral WHU-OHS, and RGB LoveDA, broadening the modalities and scene layouts shown. ScalSpectra yields clearer semantic-region separation and more coherent spatial structure than the compared remote-sensing foundation models (Section 4.5).

AuthorsOpenReview ↗

Follow-up on Author Response and Remaining Concerns

Thank you again for your constructive comments and for considering our response. As the discussion period is nearing its end, we would be grateful for a brief indication of whether our clarifications address the main concerns you raised or, if not, which specific point would benefit from further explanation. We are also happy to provide a concise summary of the corresponding revisions and supporting evidence. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.